Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create Device Models Repository Client #14863

Merged
merged 87 commits into from
May 21, 2021
Merged
Show file tree
Hide file tree
Changes from 69 commits
Commits
Show all changes
87 commits
Select commit Hold shift + click to select a range
f19be7a
first commit dmr
YoDaMa Feb 19, 2021
be96c30
rename
YoDaMa Feb 19, 2021
bdfc7fd
many changes
YoDaMa Apr 8, 2021
3f9798a
alot
YoDaMa Apr 8, 2021
8219fa9
refactor around new architecture
YoDaMa Apr 13, 2021
7e15c99
add more changes
YoDaMa Apr 13, 2021
438238f
remove tests and fix readme
YoDaMa Apr 13, 2021
da8a359
readme
YoDaMa Apr 13, 2021
b537e91
update infrastructure
YoDaMa Apr 13, 2021
117b6ba
more
YoDaMa Apr 13, 2021
f0c69e1
adding stuff
YoDaMa Apr 19, 2021
4fca643
linting
YoDaMa Apr 20, 2021
3f7c510
add launch.json to gitignore
YoDaMa Apr 20, 2021
c48dd65
fix vscode ignore
YoDaMa Apr 20, 2021
b755192
multiple
YoDaMa Apr 20, 2021
5589e21
Merge branch 'master' into device-models-tools
YoDaMa Apr 20, 2021
6e78c4e
adding api
YoDaMa Apr 20, 2021
0407787
update documentation
YoDaMa Apr 22, 2021
d20073c
example changes and bugfixes
YoDaMa Apr 24, 2021
a9ee639
refactor
YoDaMa Apr 28, 2021
84e25d9
bugfixes
YoDaMa May 4, 2021
9686d19
bugfixes
YoDaMa May 4, 2021
9650a3e
change readmes
YoDaMa May 4, 2021
bd9e9c4
fix
YoDaMa May 4, 2021
d17b3a6
update tests
YoDaMa May 13, 2021
635fe5b
update npm lock
YoDaMa May 13, 2021
6d99014
add back tests
YoDaMa May 13, 2021
7e33fd2
Fix ReadMe
YoDaMa May 14, 2021
98904a5
fix readme
YoDaMa May 14, 2021
f4e6792
update readmes
YoDaMa May 14, 2021
a393f91
addressing comments
YoDaMa May 17, 2021
76ac761
split URL to node/browser versions and add brower mapping to package.…
chradek May 17, 2021
54a6be1
Merge pull request #1 from chradek/iot-models-repo
YoDaMa May 17, 2021
513fb55
fixes
YoDaMa May 17, 2021
7207a5f
Merge branch 'device-models-tools' of https://github.com/YoDaMa/azure…
YoDaMa May 17, 2021
cc50153
updating samples and tests
YoDaMa May 18, 2021
38ff0f3
Merge remote-tracking branch 'upstream/master' into device-models-tools
YoDaMa May 18, 2021
aa8f0cd
update pnpm
YoDaMa May 18, 2021
684f954
allowinsecureConnection
YoDaMa May 18, 2021
83446b2
remove unnecessary env
YoDaMa May 18, 2021
cf9c75a
point to package
YoDaMa May 18, 2021
8326da7
readability
YoDaMa May 18, 2021
63828a0
remove old api
YoDaMa May 18, 2021
99a5ed5
comments
YoDaMa May 18, 2021
dd61713
tweaks
YoDaMa May 18, 2021
f8daa06
remove resolverError
YoDaMa May 18, 2021
bcfc92f
update resolver code
YoDaMa May 18, 2021
a6fab62
remove then
YoDaMa May 18, 2021
e34ed32
add link
YoDaMa May 18, 2021
7d06b30
fetcher changes
YoDaMa May 18, 2021
88dc580
update code
YoDaMa May 18, 2021
a2804b4
changes
YoDaMa May 18, 2021
e9ac523
fix api
YoDaMa May 19, 2021
3393208
remove en
YoDaMa May 19, 2021
18ff958
normalize
YoDaMa May 19, 2021
b006c90
fix
YoDaMa May 19, 2021
5d836c3
remove circular dependencies and fix mappings
chradek May 19, 2021
86a0a1d
Merge pull request #2 from chradek/iot-models-repo-test
YoDaMa May 19, 2021
92f6998
fix dependency
YoDaMa May 19, 2021
1702077
fix browser tst
YoDaMa May 19, 2021
c9300eb
fix links
YoDaMa May 20, 2021
1a16b3b
remove env
YoDaMa May 20, 2021
c8d840a
fix api ref link
YoDaMa May 20, 2021
ed45199
remove bad link
YoDaMa May 20, 2021
c61bdd5
fix
YoDaMa May 20, 2021
b3a9d2e
samples
YoDaMa May 20, 2021
3bc841a
remove v0 samples
YoDaMa May 20, 2021
9cd6f42
change constant
YoDaMa May 20, 2021
c2a5da1
add const unit test
YoDaMa May 20, 2021
734bda0
address comments
YoDaMa May 20, 2021
14a1861
remove todos
YoDaMa May 20, 2021
c6b2e0e
remove extend
YoDaMa May 20, 2021
8a8360d
remove how to use
YoDaMa May 21, 2021
1e3e54c
formattign
YoDaMa May 21, 2021
fd28662
adjust
YoDaMa May 21, 2021
e4e0aad
date
YoDaMa May 21, 2021
cd32118
update lockfile
YoDaMa May 21, 2021
3c0ba8f
Merge remote-tracking branch 'upstream/master' into device-models-tools
YoDaMa May 21, 2021
09500de
fix
YoDaMa May 21, 2021
3479f5f
fix stuff
YoDaMa May 21, 2021
7558cd8
first
YoDaMa May 21, 2021
626fccd
remove gitignore changes
YoDaMa May 21, 2021
96505be
remove dom
YoDaMa May 21, 2021
471acf5
add documentation
YoDaMa May 21, 2021
897c899
change constants
YoDaMa May 21, 2021
20a7b3c
add dom
YoDaMa May 21, 2021
a80e4a1
add comments for dom
YoDaMa May 21, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -88,8 +88,10 @@ examples/bin
.vs
TestResults/*
.vscode/*
**/.vscode/*
YoDaMa marked this conversation as resolved.
Show resolved Hide resolved
./**/.vscode/*


# Node #
**/node_modules/
**/cjs/
Expand Down Expand Up @@ -152,4 +154,4 @@ tsdoc-metadata.json
swagger/*.json

#autorest artifacts
code-model-*
code-model-*
68 changes: 58 additions & 10 deletions common/config/rush/pnpm-lock.yaml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

5 changes: 5 additions & 0 deletions rush.json
Original file line number Diff line number Diff line change
Expand Up @@ -661,6 +661,11 @@
"projectFolder": "sdk/digitaltwins/digital-twins-core",
"versionPolicyName": "client"
},
{
"packageName": "@azure/iot-modelsrepository",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the package name should be updated to @azure/iot-models-repository. The typical convention is to use dashes to separate words in package names (e.g. iot-device-update)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with you for the typical convention, but to follow the same naming convention for the existing models repository clients in other languages, (see python), keeping it as @azure/iot-modelsrepository will maintain cross language consistency.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we care about 'cross language consistency' for package names? The package names are already quite different for assembly-based systems like C# and Java. I don't think JS customers care about Python package naming conventions.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We got feedback from the SDK review board to use modelsrepository as the service name. We are trying to be consistent across language SDKs to make it easier to find the package/code.

I see other JS packages not using that convention eg: @azure/servicefabric

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The example is not a track 2 library. Every track 2 package in https://github.com/Azure/azure-sdk-for-js/blob/master/rush.json appears to use dash as a separator.

Do you believe that folks searching npm won't find models-respository over modelsrepository? I would imagine folks would search with a space, and most indexers break on - but would have more trouble doing prefix searches for partial matches.

"projectFolder": "sdk/iot/modelsrepository",
"versionPolicyName": "client"
},
{
"packageName": "@azure-tests/perf-ai-form-recognizer",
"projectFolder": "sdk/formrecognizer/perf-tests/ai-form-recognizer",
Expand Down
1 change: 1 addition & 0 deletions sdk/iot/modelsrepository/.npmrc
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
package-lock=false
19 changes: 19 additions & 0 deletions sdk/iot/modelsrepository/.nycrc
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
{
"include": [
"dist-esm/src/**/*.js"
],
"exclude": [
"**/*.d.ts",
"dist-esm/src/generated/*"
],
"reporter": [
"text-summary",
"html",
"cobertura"
],
"exclude-after-remap": false,
"sourceMap": true,
"produce-source-map": true,
"instrument": true,
"all": true
}
7 changes: 7 additions & 0 deletions sdk/iot/modelsrepository/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Release History

## 1.0.0-beta.1 (Unreleased)

With [#14863](https://github.com/Azure/azure-sdk-for-js/pull/14863), this is the first release of the @azure/iot-modelsrepository package.

This package contains the `ModelsRepositoryClient` to talk to the Azure Models Repository service, with initial support for getting models. Additionally, helper functions for working with DTMIs are provided.
71 changes: 71 additions & 0 deletions sdk/iot/modelsrepository/CODE_STRUCTURE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
## index.ts
YoDaMa marked this conversation as resolved.
Show resolved Hide resolved

For all typescript files there should be an entrance point. That is `index.ts`. This file serves as a translation point per-se, since it should not have any 'logic' code in it. What is exported through this file defines the API of the resolver library.
YoDaMa marked this conversation as resolved.
Show resolved Hide resolved

Though this is in a client folder, this is not really a client. It is a helper library. Because of the idiosynicatic differences between js, python, and C# (the currently implemented model repo 'clients'), C# is more like a client in that it has instantiation and a slightly different API.
YoDaMa marked this conversation as resolved.
Show resolved Hide resolved

For js (aka Node), instantiation is not common unless for larger libraries, or where it makes sense to have instances. In this case, there's not strong motivation for instantiation. So the API is simple:
YoDaMa marked this conversation as resolved.
Show resolved Hide resolved

```js
library.resolve(...)
YoDaMa marked this conversation as resolved.
Show resolved Hide resolved
```

This makes the use of our js library more convenient for users.

## resolver.ts

Contains the logic for defining the api for the `resolve` method. The implementation logic is not contained in this file.
YoDaMa marked this conversation as resolved.
Show resolved Hide resolved

The `resolve` method takes two main arguments:

- `dtmi (type: string)` - This is a user dtmi used for the dtdl the user intends to resolve. dtmi is a standard format, and if the dtmi provided does not follow the format it will be rejected.
YoDaMa marked this conversation as resolved.
Show resolved Hide resolved
- `endpoint (type:string)` - Can be a URL to a server endpoint, local or remote. Alternatively can be an _absolute_ file path, if the dtdl is stored locally. In most cases you will be interacting with the device models repository, so the endpoint will be `https://devicemodels.azure.com`, however we do not set defaults so there's no question about behavior. Simple is easier to understand!

The `resolve` method has optional parameters provided as a single object. You would use it as such:

```js
resolve(myDtmi, myEndpoint, { resolveDependencies: "disabled" });
```

Currently there is only one parameter in the optional object:

- `resolveDependencies (type: string)` - This is a useful way to get dependencies in one network call, and is recommended if you want to resolve the full dependency tree of a dtdl stored in the device model repository. These are the three options:
- `disabled`
- `enabled`
- `tryFroExpanded`

## dtmiConventions.ts

Contains methods for checking that the DTMI is valid, and to convert the DTMI to a string. This is currently private, however there are discussions around making these helper functions public parts of the API.

#### `isValidDtmi`

Validates if the provided dtmi matches the rules for a user dtmi.

#### `dtmiToPath`

Validates then converts the dtmi to a generic path.

#### `dtmiToQualifiedPath`

Validates the dtmi then converts the endpoint and dtmi to a fully qualified path. To get the `extended.json` version of a dtdl, there is a boolean parameter required.

## DTDL.ts
YoDaMa marked this conversation as resolved.
Show resolved Hide resolved

This is used to define an interface uesd in `dtmiConventions.ts`. Though it is an incomplete interface, it is just used to define the psuedo-parsing requirements.

## modelFetcherHandler.ts

This handles figuring out which fetcher to use based on the type of endpoint and the options given. It will check the endpoint to see if it is a remote URL or a local file. Then, it will pass the parameters either to the remote fetcher or the local fetcher.

## localModelFetchers.ts

This contains the local `fetcher()` method and the `recursiveFetcher()` method. The regular `fetcher()` method is called when resolverOptions are set to `disabled`, and also used as the basic unit of fetching from a directory for the `recursiveFetcher()`. The `localModelFetchers.ts` and `remoteModelFetchers.ts` are fairly similar, except for handling the formatting of the endpoint/directory, and the use of `filesystem` in the case of the `localModelFetchers`.

## remoteModelFetchers.ts

This contains the `fetcher()` method and the `recursiveFetcher()` method for fetching DTDLs from remote endpoints. It is similarly structured to `localModelFetchers.ts`. In order to perform the HTTP requests it uses the Azure `coreHttp` implementation.

## modelMetadata.ts

Performs a psuedo-parsing of a given DTDL and gives back information primarily relevant for fetching dependencies.
Loading