Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Azure IoT CA functionality #4804

Merged
merged 4 commits into from
Nov 7, 2017
Merged

Azure IoT CA functionality #4804

merged 4 commits into from
Nov 7, 2017

Conversation

andrew-buckley
Copy link
Contributor

@andrew-buckley andrew-buckley commented Nov 1, 2017


This checklist is used to make sure that common guidelines for a pull request are followed.

General Guidelines

  • The PR has modified HISTORY.rst describing any customer-facing, functional changes. Note that this does not include changes only to help content. (see Modifying change log).

Command Guidelines

  • Each command and parameter has a meaningful description.
  • Each new command has a test.

(see Authoring Command Modules)

Sorry, something went wrong.

Copy link
Member

@tjprescott tjprescott left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall LGTM. A couple comments. Also, are you the new POC for IoT commands?

.gitignore Outdated
*.key
*.cer
*.pem
*.pvk
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are these added?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is so that test certs that are created in the repo during tests aren't accidentally checked in if they fail to get deleted. I can simplify this to only 2 types instead of 4.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are cer files and pem files checked in with batch and keyvault tests, which they treat as test resource. Better not to exclude these file at the global level. You can add .gitignore to iot folder to narrow the scope.

23:56 $ find . -name '*.cer'
./src/command_modules/azure-cli-batch/azure/cli/command_modules/batch/tests/data/batchtest.cer
./src/command_modules/azure-cli-network/azure/cli/command_modules/network/tests/test-root-cert.cer

23:56 $ find . -name '*.pem'
./env/lib/python3.5/site-packages/certifi/cacert.pem
./env/lib/python3.5/site-packages/certifi/old_root.pem
./env/lib/python3.5/site-packages/certifi/weak.pem
./env/lib/python3.5/site-packages/pip/_vendor/requests/cacert.pem
./src/azure-cli-core/azure/cli/core/tests/sp_cert.pem
./src/command_modules/azure-cli-keyvault/azure/cli/command_modules/keyvault/tests/import_pem_encrypted_pwd_123.pem
./src/command_modules/azure-cli-keyvault/azure/cli/command_modules/keyvault/tests/import_pem_encrypted_pwd_1234.pem
./src/command_modules/azure-cli-keyvault/azure/cli/command_modules/keyvault/tests/import_pem_plain.pem
./src/command_modules/azure-cli-keyvault/azure/cli/command_modules/keyvault/tests/mydomain.test.encrypted.pem
./src/command_modules/azure-cli-keyvault/azure/cli/command_modules/keyvault/tests/mydomain.test.pem
./src/command_modules/azure-cli-role/azure/cli/command_modules/role/tests/cert.pem

return certificate


def _create_test_cert(cert_file, key_file, subject, valid_days, serial_number):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is this used for? If only for tests, consider moving it into your test directory so that it doesn't find its way into a command some day.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this specific function is only used in tests, I'll separate these out.

custom_path.format('iot_hub_certificate_gen_code'), certificate_factory)
cli_command(__name__, 'iot hub certificate verify', custom_path.format('iot_hub_certificate_verify'),
certificate_factory)
cli_command(__name__, 'iot hub certificate update', custom_path.format('iot_hub_certificate_update'),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be a generic update.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you mean by "generic update"?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gotcha, is this a valid scenario for this generic update command? In this case, the Certificate object is created by providing a friendly name and a certificate string. Once it's created the ONLY thing that can be updated is the entire certificate (meaning uploading an entirely new certificate string). The name cannot be 'updated' nor can any of the properties. Please correct me if I'm wrong, but does the generic update command simply provide a way to craft your update to an object before writing it back to the server?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It wasn't completely clear to me from the description in the doc.

def iot_hub_certificate_factory(_):
from azure.cli.core.commands.client_factory import get_mgmt_service_client
from azure.mgmt.iothub.iot_hub_client import IotHubClient
return (get_mgmt_service_client(IotHubClient).iot_hub_resource, get_mgmt_service_client(IotHubClient).certificates)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can simply return get_mgmt_service_client(IotHubClient) and then based on what you need in your commands, reference .iot_hub_resource or .certificates as needed. It would make your custom commands much easier to read.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, let me investigate this!

Copy link
Member

@derekbekoe derekbekoe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You need to change the version in setup.py. CI will fail because of this.

cert_list = client[1].list_by_iot_hub(resource_group_name, hub_name)
for cert in cert_list.value:
if cert.name == certificate_name:
raise CLIError('Certificate \'{0}\' already exists. Use \'iot hub certificate update\' to update an existing certificate.'.format(certificate_name))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would just suggest using double quotes CLIError("...") then you don't need to escape the single quotes and it's clearer.

0.1.15
++++++
* Adds support for certificate authorities (CA) and certificate chains.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need to update the version, only 0.1.13 has been released.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah yes so move the changelog to 0.1.14.

try:
certificate = certificate.decode("utf-8")
except UnicodeError:
certificate = base64.b64encode(certificate).decode("utf-8")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to confirm, we don't want to encode the certificate with base64 if all the characters are valid unicode?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that's correct.

@williexu
Copy link
Contributor

williexu commented Nov 1, 2017

Also, include the recordings for your tests please.

@andrew-buckley
Copy link
Contributor Author

@tjprescott I would note my manager @iluican as the POC for IoT commands going forward as I won't be the only one contributing from our team. Working on addressing your comments now.

.gitignore Outdated
*.key
*.cer
*.pem
*.pvk
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are cer files and pem files checked in with batch and keyvault tests, which they treat as test resource. Better not to exclude these file at the global level. You can add .gitignore to iot folder to narrow the scope.

23:56 $ find . -name '*.cer'
./src/command_modules/azure-cli-batch/azure/cli/command_modules/batch/tests/data/batchtest.cer
./src/command_modules/azure-cli-network/azure/cli/command_modules/network/tests/test-root-cert.cer

23:56 $ find . -name '*.pem'
./env/lib/python3.5/site-packages/certifi/cacert.pem
./env/lib/python3.5/site-packages/certifi/old_root.pem
./env/lib/python3.5/site-packages/certifi/weak.pem
./env/lib/python3.5/site-packages/pip/_vendor/requests/cacert.pem
./src/azure-cli-core/azure/cli/core/tests/sp_cert.pem
./src/command_modules/azure-cli-keyvault/azure/cli/command_modules/keyvault/tests/import_pem_encrypted_pwd_123.pem
./src/command_modules/azure-cli-keyvault/azure/cli/command_modules/keyvault/tests/import_pem_encrypted_pwd_1234.pem
./src/command_modules/azure-cli-keyvault/azure/cli/command_modules/keyvault/tests/import_pem_plain.pem
./src/command_modules/azure-cli-keyvault/azure/cli/command_modules/keyvault/tests/mydomain.test.encrypted.pem
./src/command_modules/azure-cli-keyvault/azure/cli/command_modules/keyvault/tests/mydomain.test.pem
./src/command_modules/azure-cli-role/azure/cli/command_modules/role/tests/cert.pem

@@ -5,6 +5,9 @@

from azure.cli.core.help_files import helps

certificate_help = ('For a detailed explanation of CA certificates in Azure IoT Hub, '
'see https://docs.microsoft.com/en-us/azure/iot-hub/iot-hub-x509ca-overview')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You don't need the parenthesis.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couldn't get this to work without them

@@ -23,16 +23,31 @@ def get_device_id_completion_list(prefix, action, parsed_args,
completer=get_resource_name_completion_list('Microsoft.Devices/IotHubs'),
help='IoT Hub name.')

etag_type = CliArgumentType(
None,
help='Entity Tag (etag) of the object.')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Key this on the same line to make code condense.

None,
help='Entity Tag (etag) of the object.')


register_cli_argument('iot hub', 'hub_name', hub_name_type, options_list=('--name', '-n'),
id_part='name')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above. The line limitation is 120.


helps['iot hub certificate create'] = """
type: command
short-summary: Create/upload an Azure IoT Hub certificate.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this command iot hub certificate create when --path to file containing the certificate is required?
Looks like this should be iot hub certificate upload?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is a bit strange. The command creates a Certificate object in a users IoT Hub. One of the parameters for creation is a Unicode string that represents the certificate. It's not truly uploading a file, but rather we are extracting the file contents if it's valid Unicode and using that as the "certificate" parameter, or base64 encoding the contents and using that string as the param. Additionally, we were trying to keep command name parity with the other operations we own.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay I understand.

register_cli_argument('iot hub {}'.format(subgroup), 'hub_name', options_list=('--hub-name',))

register_cli_argument('iot device', 'hub_name', hub_name_type)

register_cli_argument('iot', 'device_id', options_list=('--device-id', '-d'), help='Device Id.',
completer=get_device_id_completion_list)

# Arguments for 'iot hub certificate' group
register_cli_argument('iot hub certificate', 'certificate_path', options_list=('--path', '-p'),
help='The path to the file containing the certificate.')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be good to add a File completer to this.

@andrew-buckley
Copy link
Contributor Author

@derekbekoe is the version in setup.py supposed to match the most recently released version or the highest one in HISTORY.rst?

@derekbekoe
Copy link
Member

0.1.13 is the latest released right now so in HISTORY.rst, the top entry should be 0.1.14 and in setup.py, the version should be 0.1.14.
Hope that clarifies.

@andrew-buckley
Copy link
Contributor Author

@williexu @troydai @derekbekoe seems like 9097.3 failure is unrelated to this work. However. 9097.4 seems to be a linting failure that I'm not able to repro locally on python 3.6.

@derekbekoe
Copy link
Member

I've restarted 9097.3. I think it's a flaky test.
For .4, we used to skip E126 checks but now they are required.
Rebase/merge your changes with the latest from the dev branch then you should be able to repro locally. You may not have commit 63ee8481 locally.

@derekbekoe
Copy link
Member

I suspect the error for .3 is due to one of the changes in your help.py as it happened again.
It could be the single quotes you have for some long-summary.

@andrew-buckley
Copy link
Contributor Author

andrew-buckley commented Nov 6, 2017

After doing the rebase I'm no longer able to run tests. <clone_root>/src/scripts no longer contains the run_tests and check_style scripts. Seems like they now live in env/scripts. I made the style changes and the linter passes, however many tests seem to be failing suddenly. Using the setup instructions here which had previously been working for me: https://github.com/Azure/azure-cli/blob/dev/doc/configuring_your_machine.md

@andrew-buckley
Copy link
Contributor Author

Got the tests to pass by going to an earlier python version. Seems like the scripts have been moved though.

@williexu
Copy link
Contributor

williexu commented Nov 7, 2017

@andrew-buckley That should be fine. The scripts are added to your python environment upon running python dev_setup.py

@derekbekoe
Copy link
Member

After pulling the latest changes, run python scripts/dev_setup.py again to get the latest dependencies.

@andrew-buckley
Copy link
Contributor Author

Hm, I ran through those steps (https://github.com/Azure/azure-cli/blob/dev/doc/configuring_your_machine.md) a few times and again after going down to 3.5.4 but still didn't have any luck.

@andrew-buckley
Copy link
Contributor Author

Seems like *.3 is still failing with the same error. I tried removing the single quotes from the help text but it doesn't seem like that helped. Is there a better way to narrow down this problem? Maybe locally, so that I don't have to push updates and wait for a build job to run in order to do trial and error?

@troydai
Copy link
Contributor

troydai commented Nov 7, 2017

@andrew-buckley our doc is a bit behind. sorry for that. To run the style check locally:

  1. set up virtualenv
  2. run dev_setup.py
  3. run check_style --module iot

@williexu
Copy link
Contributor

williexu commented Nov 7, 2017

@andrew-buckley after the steps @troydai listed, run run_tests --module <module> --test <filename>.[<classname>.[<testmethodname>]]
Ex, for one of the tests that are failing in 9114.3, you'll want to run:
run_tests --module find --test test_find.SearchIndexTest.test_search_index

@andrew-buckley
Copy link
Contributor Author

@williexu thanks for that. I tried those steps but they're passing for me. Although I'm running python 3.5, I'll try on 2.7 as it seems that's the version it's failing for.

@andrew-buckley
Copy link
Contributor Author

Having a bit of trouble getting setup going with python 2.7 following the instructions in the doc as well as those described above by @troydai.

@williexu
Copy link
Contributor

williexu commented Nov 7, 2017

Which part? did you create and activate a virtual environment using python 2?
The other steps should be the same.

@andrew-buckley
Copy link
Contributor Author

I'm rather new to python so apologies if I'm clearly doing something wrong here. I'm having trouble creating the virtual environment. I tried uninstalling python 3.5 from my machine:

F:\Projects\Ecosystem\azure-cli-fork>python --version
Python 2.7.10

F:\Projects\Ecosystem\azure-cli-fork>pip --version
pip 9.0.1 from C:\Python27\lib\site-packages (python 2.7)

F:\Projects\Ecosystem\azure-cli-fork>pip install virtualenv
Collecting virtualenv
  Using cached virtualenv-15.1.0-py2.py3-none-any.whl
Installing collected packages: virtualenv
Successfully installed virtualenv-15.1.0

F:\Projects\Ecosystem\azure-cli-fork>python -m virtualenv env
New python executable in F:\Projects\Ecosystem\azure-cli-fork\env\Scripts\python.exe
Installing setuptools, pip, wheel...
  Complete output from command F:\Projects\Ecosyste...v\Scripts\python.exe - setuptools pip wheel:
  Traceback (most recent call last):
  File "<stdin>", line 7, in <module>
  File "C:\Python27\lib\site-packages\virtualenv_support\pip-9.0.1-py2.py3-none-any.whl\pip\__init__.py", line 21, in <module>
  File "C:\Python27\lib\site-packages\virtualenv_support\pip-9.0.1-py2.py3-none-any.whl\pip\_vendor\requests\__init__.py", line 62, in <module>
  File "C:\Python27\lib\site-packages\virtualenv_support\pip-9.0.1-py2.py3-none-any.whl\pip\_vendor\requests\packages\__init__.py", line 29, in <module>
  File "F:\Projects\Ecosystem\azure-cli-fork\env\lib\site-packages\urllib3\__init__.py", line 8, in <module>
    from .connectionpool import (
  File "F:\Projects\Ecosystem\azure-cli-fork\env\lib\site-packages\urllib3\connectionpool.py", line 7, in <module>
    from socket import error as SocketError, timeout as SocketTimeout
  File "C:\Python27\Lib\socket.py", line 47, in <module>
    import _socket
ImportError: Module use of python35.dll conflicts with this version of Python.
----------------------------------------
...Installing setuptools, pip, wheel...done.
Traceback (most recent call last):
  File "C:\Python27\lib\runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "C:\Python27\lib\runpy.py", line 72, in _run_code
    exec code in run_globals
  File "C:\Python27\lib\site-packages\virtualenv.py", line 2328, in <module>
    main()
  File "C:\Python27\lib\site-packages\virtualenv.py", line 713, in main
    symlink=options.symlink)
  File "C:\Python27\lib\site-packages\virtualenv.py", line 945, in create_environment
    download=download,
  File "C:\Python27\lib\site-packages\virtualenv.py", line 901, in install_wheel
    call_subprocess(cmd, show_stdout=False, extra_env=env, stdin=SCRIPT)
  File "C:\Python27\lib\site-packages\virtualenv.py", line 797, in call_subprocess
    % (cmd_desc, proc.returncode))
OSError: Command F:\Projects\Ecosyste...v\Scripts\python.exe - setuptools pip wheel failed with error code 1

F:\Projects\Ecosystem\azure-cli-fork>

@andrew-buckley
Copy link
Contributor Author

FYI I'm attempting to install and use virtualenv as it seems like venv only ships with python 3.3+

@troydai
Copy link
Contributor

troydai commented Nov 7, 2017

the virtualenv should be working on both python 2 and 3. Can you install the latest python from here and try the virtualenv again: https://www.python.org/downloads/.

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
…mands.py. Moves test utilities. Adds a file completer.
@andrew-buckley
Copy link
Contributor Author

@williexu please take a look when you can. All checks have passed.

@williexu
Copy link
Contributor

williexu commented Nov 7, 2017

LGTM

@williexu williexu merged commit 55b7b73 into Azure:dev Nov 7, 2017
LukaszStem pushed a commit to LukaszStem/azure-cli that referenced this pull request Nov 9, 2017
vmss: support basic tier of vms (Azure#4847)

Azure IoT CA functionality (Azure#4804)

* Adds support for X.509 Certificates in IoT Hub.

* Modifies release notes and changes help link.

* Cleans up .gitignore, and HISTORY.rst. Passes a single factory in commands.py. Moves test utilities. Adds a file completer.

* Updates style after rebase based on additional linter restrictions.

Reserved Instance cli public PR (Azure#4838)

Handle dashes in extension names better (Azure#4839)

Fix `az aks get-credentials` on Windows (Azure#4762)

* Fix `az aks get-credentials` on Windows

* Move k8s file merge logic to helper function

appservice: support assign managed service identity to webapp/functionapp (Azure#4837)

Extension examples small improvement (Azure#4852)

Fixing appservice list-locations (Azure#4846)

Add extension name to telemetry and UA header (Azure#4854)

Fix Azure#4825. (Azure#4853)

Extensions `az extension add` --yes param as flag (Azure#4858)

Fix issue Azure#2752. (Azure#4859)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants