Skip to content
This repository has been archived by the owner on Nov 4, 2024. It is now read-only.

Commit

Permalink
Merge persistence work into master (microsoft#1482)
Browse files Browse the repository at this point in the history
* Persistent analysis, part I (microsoft#1224)

* Remove old qualified name

* Node storage

* Class and scope to use AST map

* Library analysis

* Fix SO

* Keep small AST with imports

* AST reduction

* Final field

* Initial

* Reload

* Ignore post-final requests

* Drop AST

* Remove local variables

* Test fixes

* Fix overload match

* Tests

* Add locks

* Remove local variables

* Drop file content to save memory

* Cache PEP hints

* Recreate AST

* Fix specialization

* Fix locations

* usings

* Test fixes

* Add options to keep data in memory

* Fix test

* Fix lambda parameters

* Fix argument set
Fix global scope node

* Fix overload doc

* Fix stub merge errors

* Fix async issues

* Undo some changes

* Fix test

* Fix race condition

* Partial

* Models and views

* Restore log null checks

* Fix merge conflict

* Fix merge issue

* Null check

* Partial

* Partial

* Partial

* Fix test

* Partial

* Partial

* First test

* Baseline comparison

* Builtins

* Partial

* Type fixes

* Fix type names, part I

* Qualified name

* Properly write variables

* Partial

* Construct module from model

* Test

* Variable creations

* Factories

* Factories

* Split construction

* Restore

* Save builtins

* Test passes

* Qualified name

* Better export detection

* Test fixes

* More consistent qualified names

* Sys test

* Demo

* Complete sys write/read

* Partial

* Partial

* Test staility

* Perf bug

* Baseline, remove debug code, deactivate db

* Test fixes

* Test fix

* Simplify a bit

* Baselines and use : separator

* Baselines

* PR feedback

* Merge master

* Remove registry reference

* PR feedback

* Merge master into database branch (microsoft#1312)

* Remove old qualified name

* Node storage

* Class and scope to use AST map

* Library analysis

* Fix SO

* Keep small AST with imports

* AST reduction

* Final field

* Initial

* Reload

* Ignore post-final requests

* Drop AST

* Remove local variables

* Test fixes

* Fix overload match

* Tests

* Add locks

* Remove local variables

* Drop file content to save memory

* Cache PEP hints

* Recreate AST

* Fix specialization

* Fix locations

* usings

* Test fixes

* Add options to keep data in memory

* Fix test

* Fix lambda parameters

* Fix argument set
Fix global scope node

* Fix overload doc

* Fix stub merge errors

* Fix async issues

* Undo some changes

* Fix test

* Fix race condition

* Partial

* Models and views

* Restore log null checks

* Fix merge conflict

* Fix merge issue

* Null check

* Partial

* Partial

* Partial

* Fix test

* Partial

* Partial

* First test

* Baseline comparison

* Builtins

* Partial

* Type fixes

* Fix type names, part I

* Qualified name

* Properly write variables

* Partial

* Construct module from model

* Test

* Variable creations

* Factories

* Factories

* Split construction

* Restore

* Save builtins

* Test passes

* Qualified name

* Better export detection

* Test fixes

* More consistent qualified names

* Sys test

* Demo

* Complete sys write/read

* Partial

* Partial

* Test staility

* Perf bug

* Baseline, remove debug code, deactivate db

* Test fixes

* Test fix

* Simplify a bit

* Baselines and use : separator

* Baselines

* PR feedback

* Merge master

* Remove registry reference

* PR feedback

* Avoid empty node in empty global scope (microsoft#1226)

* Add null checks

* Avoid null node in empty global scope

* Remove unrelated change

* Lock content since it is locked in other cases (microsoft#1227)

* Fix addBrackets setting (microsoft#1241)

* Fix addBrackets

* Remove unrelated change

* Usings

* Handle library analysis better (microsoft#1246)

* Trying to augment argument set with Evaluator and Expression context (microsoft#1259)

* Moving anyStr to its own method to prevent FromTypeVar from being called with an ArgumentSet that has no context

* Trying to augment argument set with context wherever possible

* Setting expression in argument set, removing null

* Renaming to WithoutContext and updating comment

* Check for null args in PythonFunctionOverload

* When returning in init method, report a diagnostic error (microsoft#1261)

* When returning in init method, report a diagnostic error

* Adding check and tests for returning None before adding error for a return statement in the init function

* Cleaning up tests

* Give Diagnostic messages on improper usage of Generic (microsoft#1248)

* Adding diagnostic message for improper use Generic.

* Adding tests to make sure no diagnostics on valid uses of Generic

* Adding diagnostic error on binary operations with incompatible types (microsoft#1254)

* Adding diagnostic error for binary operations with incompatible types - e.g 5 + 'str'

* When NewType is called and the first argument is not a string, make a diagnostic message (microsoft#1260)

* Adding diagnostic error if the user calls NewType with the first arg not of string type

* Fix typo in TROUBLESHOOTING.md (microsoft#1285)

* Multiple analysis fixes (microsoft#1297)

Multiple analysis fixes.
- Fixes microsoft#1151: Reliable way to determine final analysis of a module
- Fixes microsoft#1201: NRE in SymbolCollector.AddProperty
- Fixes microsoft#1228: NRE in TryFindMissingDependencies
- Fixes microsoft#1294: Handle valid case for reloading AST
- Fixes microsoft#1295: FindReferences doesn't work on package init members
- Fixes microsoft#1296: DependencyResolver graph misses intermediate imports
- Part of the fix for microsoft#1174: All references are not listed when using ms language server (fixes in analysis itself are also required for this one)

* Rework search path resolution (microsoft#1289)

* first working path classfier, breaks existing path code for now

* inherit Comparer instead of implementing both

* don't do path depth comparison, preserve user ordering as before

* switch to using new path classification code in main module resolution

* fix typo

* clean up comment about paths in Server.cs

* use String.Split instead of regex when parsing script output

* test ordering of user provided paths

* add new normalize and trim helper, check preconditions as debug asserts rather than commenting on them

* use Split extension, don't MaybeEnumerate paths

* No diagnostic message when Generic is called with no args (microsoft#1305)

* No diagnostic message when Generic is called with no args

* When a class inherits from something that is not a class, give a diagnostic error (microsoft#1277)

* When analyzing class definitions, give diagnostic if bases are not all class types

* PR feedback

* Treat persistent module as regular specialized (microsoft#1334)

* Remove old qualified name

* Node storage

* Class and scope to use AST map

* Library analysis

* Fix SO

* Keep small AST with imports

* AST reduction

* Final field

* Initial

* Reload

* Ignore post-final requests

* Drop AST

* Remove local variables

* Test fixes

* Fix overload match

* Tests

* Add locks

* Remove local variables

* Drop file content to save memory

* Cache PEP hints

* Recreate AST

* Fix specialization

* Fix locations

* usings

* Test fixes

* Add options to keep data in memory

* Fix test

* Fix lambda parameters

* Fix argument set
Fix global scope node

* Fix overload doc

* Fix stub merge errors

* Fix async issues

* Undo some changes

* Fix test

* Fix race condition

* Partial

* Models and views

* Restore log null checks

* Fix merge conflict

* Fix merge issue

* Null check

* Partial

* Partial

* Partial

* Fix test

* Partial

* Partial

* First test

* Baseline comparison

* Builtins

* Partial

* Type fixes

* Fix type names, part I

* Qualified name

* Properly write variables

* Partial

* Construct module from model

* Test

* Variable creations

* Factories

* Factories

* Split construction

* Restore

* Save builtins

* Test passes

* Qualified name

* Better export detection

* Test fixes

* More consistent qualified names

* Sys test

* Demo

* Complete sys write/read

* Partial

* Partial

* Test staility

* Perf bug

* Baseline, remove debug code, deactivate db

* Test fixes

* Test fix

* Simplify a bit

* Baselines and use : separator

* Baselines

* PR feedback

* Merge master

* Remove registry reference

* PR feedback

* PR feedback

* Restore persistence + update test

* Better handle persistent module in dependencies

* Undo

* Fix merge issue

* Remove debug code

* Store location of module members in the database  (microsoft#1339)

* Remove old qualified name

* Node storage

* Class and scope to use AST map

* Library analysis

* Fix SO

* Keep small AST with imports

* AST reduction

* Final field

* Initial

* Reload

* Ignore post-final requests

* Drop AST

* Remove local variables

* Test fixes

* Fix overload match

* Tests

* Add locks

* Remove local variables

* Drop file content to save memory

* Cache PEP hints

* Recreate AST

* Fix specialization

* Fix locations

* usings

* Test fixes

* Add options to keep data in memory

* Fix test

* Fix lambda parameters

* Fix argument set
Fix global scope node

* Fix overload doc

* Fix stub merge errors

* Fix async issues

* Undo some changes

* Fix test

* Fix race condition

* Partial

* Models and views

* Restore log null checks

* Fix merge conflict

* Fix merge issue

* Null check

* Partial

* Partial

* Partial

* Fix test

* Partial

* Partial

* First test

* Baseline comparison

* Builtins

* Partial

* Type fixes

* Fix type names, part I

* Qualified name

* Properly write variables

* Partial

* Construct module from model

* Test

* Variable creations

* Factories

* Factories

* Split construction

* Restore

* Save builtins

* Test passes

* Qualified name

* Better export detection

* Test fixes

* More consistent qualified names

* Sys test

* Demo

* Complete sys write/read

* Partial

* Partial

* Test staility

* Perf bug

* Baseline, remove debug code, deactivate db

* Test fixes

* Test fix

* Simplify a bit

* Baselines and use : separator

* Baselines

* PR feedback

* Merge master

* Remove registry reference

* PR feedback

* PR feedback

* Restore persistence + update test

* Better handle persistent module in dependencies

* Undo

* Add location converter abstraction

* Store member location

* Basic locations test

* Navigation

* Add test

* Update baselines

* Port changes from dbtype

* Multiple fixes and additional tests for persistence (microsoft#1372)

* Remove old qualified name

* Node storage

* Class and scope to use AST map

* Library analysis

* Fix SO

* Keep small AST with imports

* AST reduction

* Final field

* Initial

* Reload

* Ignore post-final requests

* Drop AST

* Remove local variables

* Test fixes

* Fix overload match

* Tests

* Add locks

* Remove local variables

* Drop file content to save memory

* Cache PEP hints

* Recreate AST

* Fix specialization

* Fix locations

* usings

* Test fixes

* Add options to keep data in memory

* Fix test

* Fix lambda parameters

* Fix argument set
Fix global scope node

* Fix overload doc

* Fix stub merge errors

* Fix async issues

* Undo some changes

* Fix test

* Fix race condition

* Partial

* Models and views

* Restore log null checks

* Fix merge conflict

* Fix merge issue

* Null check

* Partial

* Partial

* Partial

* Fix test

* Partial

* Partial

* First test

* Baseline comparison

* Builtins

* Partial

* Type fixes

* Fix type names, part I

* Qualified name

* Properly write variables

* Partial

* Construct module from model

* Test

* Variable creations

* Factories

* Factories

* Split construction

* Restore

* Save builtins

* Test passes

* Qualified name

* Better export detection

* Test fixes

* More consistent qualified names

* Sys test

* Demo

* Complete sys write/read

* Partial

* Partial

* Test staility

* Perf bug

* Baseline, remove debug code, deactivate db

* Test fixes

* Test fix

* Simplify a bit

* Baselines and use : separator

* Baselines

* PR feedback

* Merge master

* Remove registry reference

* PR feedback

* PR feedback

* Restore persistence + update test

* Better handle persistent module in dependencies

* Undo

* Add location converter abstraction

* Store member location

* Fix merge issue

* Basic locations test

* Navigation

* Add test

* Update baselines

* Type restore - initial

* Remove debug code

* Partial

* Fix stub merge

* Various model fixes

* Improve module handling

* Qualified name improvements

* Fix unbound case

* Improve stub merge

* Fix qualified names of typing

* Handle stub-only modules

* Add tests for io, re and sys

* Better handle named tuple

* Handle module circular references

* Handle else in platform and version clauses + handle it in symbol collector

* Formatting

* Fix walk of multi-level if statement

* Unify package search in imports

* Fix tests

* Undo change

* Port changes from dbtype

* Partial

* Named tuple support

* Baseline updates

* Debug code

* Support types

* Properly compare class member declaring type

* Nested classes and functions persistence

* Undo debug

* Fix numpy restore

* Baselines

* Fix tests

* Update AnyStr test reflecting changes to AnyStr behavior

* Exclude baselines from git diff

* Fix gitattr

* Move git setting to root

* Try no path

* Add RDT count to the analysis_complete event (microsoft#1396)

* Add RDT count to the analysis_complete event

* Use property instead of method

* Test fixes

* Undo change

* Additional stub merge fixes

* Baseline updates

* Fix goto def

* Protect specific type creation

* Track documentaton source

* More reliable tests + simplification

* Prevent crashes when the AST happens to be null (microsoft#1405)

* Remove all references to dropped modules and force GC on reload (microsoft#1402)

* First attempt at ensuring moduled dropped during reload are GC'd

* Make sure AST is set after content is cleared

* More tests passing

* ModuleWalker updates

* Also don't merge against specialized modules, do builtin _astMap reset before reanalyzing instead of after any analysis

* Don't call dispose directly, it's done elsewhere and has no effect

* Force GC via normal session instead of waiting for a hardcoded time

* Un-refactor function since it's not being called externally anymore

* Formatting/usings

* PR feedback

* Undo IsBuiltin, some things continued to be referenced with that changed

* Move reload logic down into PythonAnalyzer, remove added public inferface items

* Mode AddReference to PythonType as override

* Remove dead code, make all ResetAnalyzer calls full

* Typo

* Cleanup

* Basic classification

* Fix merge error

* Module unique id fixes

* Stricted check to save analysis

* Revert "Fix tests"

This reverts commit 247a8c3.

* Revert "Unify package search in imports"

This reverts commit 67fed10.

* Don't clear scope variables with inner classes

* Fix typo

* Add persistence tests for 60+ system modules (microsoft#1429)

* Remove old qualified name

* Node storage

* Class and scope to use AST map

* Library analysis

* Fix SO

* Keep small AST with imports

* AST reduction

* Final field

* Initial

* Reload

* Ignore post-final requests

* Drop AST

* Remove local variables

* Test fixes

* Fix overload match

* Tests

* Add locks

* Remove local variables

* Drop file content to save memory

* Cache PEP hints

* Recreate AST

* Fix specialization

* Fix locations

* usings

* Test fixes

* Add options to keep data in memory

* Fix test

* Fix lambda parameters

* Fix argument set
Fix global scope node

* Fix overload doc

* Fix stub merge errors

* Fix async issues

* Undo some changes

* Fix test

* Fix race condition

* Partial

* Models and views

* Restore log null checks

* Fix merge conflict

* Fix merge issue

* Null check

* Partial

* Partial

* Partial

* Fix test

* Partial

* Partial

* First test

* Baseline comparison

* Builtins

* Partial

* Type fixes

* Fix type names, part I

* Qualified name

* Properly write variables

* Partial

* Construct module from model

* Test

* Variable creations

* Factories

* Factories

* Split construction

* Restore

* Save builtins

* Test passes

* Qualified name

* Better export detection

* Test fixes

* More consistent qualified names

* Sys test

* Demo

* Complete sys write/read

* Partial

* Partial

* Test staility

* Perf bug

* Baseline, remove debug code, deactivate db

* Test fixes

* Test fix

* Simplify a bit

* Baselines and use : separator

* Baselines

* PR feedback

* Merge master

* Remove registry reference

* PR feedback

* PR feedback

* Restore persistence + update test

* Better handle persistent module in dependencies

* Undo

* Add location converter abstraction

* Store member location

* Fix merge issue

* Basic locations test

* Navigation

* Add test

* Update baselines

* Type restore - initial

* Remove debug code

* Partial

* Fix stub merge

* Various model fixes

* Improve module handling

* Qualified name improvements

* Fix unbound case

* Improve stub merge

* Fix qualified names of typing

* Handle stub-only modules

* Add tests for io, re and sys

* Better handle named tuple

* Handle module circular references

* Handle else in platform and version clauses + handle it in symbol collector

* Formatting

* Fix walk of multi-level if statement

* Unify package search in imports

* Fix tests

* Undo change

* Port changes from dbtype

* Partial

* Named tuple support

* Baseline updates

* Debug code

* Support types

* Properly compare class member declaring type

* Nested classes and functions persistence

* Undo debug

* Fix numpy restore

* Baselines

* Fix tests

* Update AnyStr test reflecting changes to AnyStr behavior

* Exclude baselines from git diff

* Fix gitattr

* Move git setting to root

* Try no path

* Test fixes

* Undo change

* Additional stub merge fixes

* Baseline updates

* Fix goto def

* Protect specific type creation

* Track documentaton source

* More reliable tests + simplification

* Typo

* Cleanup

* Basic classification

* Fix merge error

* Module unique id fixes

* Stricted check to save analysis

* Revert "Fix tests"

This reverts commit 247a8c3.

* Revert "Unify package search in imports"

This reverts commit 67fed10.

* Don't clear scope variables with inner classes

* Fix typo

* Many new tests

* Fix collections test

* Fix CTypes

* Initial

* Update test

* Merge issues

* Fix CTypes again

* Fix null bases

* Tell between class members with/without self better

* TypeVar support

* Add bound/covariant

* Fix Enum reassignments

* Fix Random

* Fix import * over local declarations (Socket)

* Move interface

* Fix reference search

* Enable asyncio test

* More tests

* Enable dataclasses

* Add inspect and gzip

* More tests

* Add setting

* Add handling of import position relative to the variable

* Caching level changes

* Update baselines

* Test update (we no longer reassign classes)

* Fix typing tests

* Update baseline

* Fix missing keys issue (microsoft#1451)

* Typevar fix

* Persistence of generics and named tuples (microsoft#1459)

* Remove old qualified name

* Node storage

* Class and scope to use AST map

* Library analysis

* Fix SO

* Keep small AST with imports

* AST reduction

* Final field

* Initial

* Reload

* Ignore post-final requests

* Drop AST

* Remove local variables

* Test fixes

* Fix overload match

* Tests

* Add locks

* Remove local variables

* Drop file content to save memory

* Cache PEP hints

* Recreate AST

* Fix specialization

* Fix locations

* usings

* Test fixes

* Add options to keep data in memory

* Fix test

* Fix lambda parameters

* Fix argument set
Fix global scope node

* Fix overload doc

* Fix stub merge errors

* Fix async issues

* Undo some changes

* Fix test

* Fix race condition

* Partial

* Models and views

* Restore log null checks

* Fix merge conflict

* Fix merge issue

* Null check

* Partial

* Partial

* Partial

* Fix test

* Partial

* Partial

* First test

* Baseline comparison

* Builtins

* Partial

* Type fixes

* Fix type names, part I

* Qualified name

* Properly write variables

* Partial

* Construct module from model

* Test

* Variable creations

* Factories

* Factories

* Split construction

* Restore

* Save builtins

* Test passes

* Qualified name

* Better export detection

* Test fixes

* More consistent qualified names

* Sys test

* Demo

* Complete sys write/read

* Partial

* Partial

* Test staility

* Perf bug

* Baseline, remove debug code, deactivate db

* Test fixes

* Test fix

* Simplify a bit

* Baselines and use : separator

* Baselines

* PR feedback

* Merge master

* Remove registry reference

* PR feedback

* PR feedback

* Restore persistence + update test

* Better handle persistent module in dependencies

* Undo

* Add location converter abstraction

* Store member location

* Fix merge issue

* Basic locations test

* Navigation

* Add test

* Update baselines

* Type restore - initial

* Remove debug code

* Partial

* Fix stub merge

* Various model fixes

* Improve module handling

* Qualified name improvements

* Fix unbound case

* Improve stub merge

* Fix qualified names of typing

* Handle stub-only modules

* Add tests for io, re and sys

* Better handle named tuple

* Handle module circular references

* Handle else in platform and version clauses + handle it in symbol collector

* Formatting

* Fix walk of multi-level if statement

* Unify package search in imports

* Fix tests

* Undo change

* Port changes from dbtype

* Partial

* Named tuple support

* Baseline updates

* Debug code

* Support types

* Properly compare class member declaring type

* Nested classes and functions persistence

* Undo debug

* Fix numpy restore

* Baselines

* Fix tests

* Update AnyStr test reflecting changes to AnyStr behavior

* Exclude baselines from git diff

* Fix gitattr

* Move git setting to root

* Try no path

* Test fixes

* Undo change

* Additional stub merge fixes

* Baseline updates

* Fix goto def

* Protect specific type creation

* Track documentaton source

* More reliable tests + simplification

* Typo

* Cleanup

* Basic classification

* Fix merge error

* Module unique id fixes

* Stricted check to save analysis

* Revert "Fix tests"

This reverts commit 247a8c3.

* Revert "Unify package search in imports"

This reverts commit 67fed10.

* Don't clear scope variables with inner classes

* Fix typo

* Many new tests

* Fix collections test

* Fix CTypes

* Initial

* Update test

* Merge issues

* Fix CTypes again

* Fix null bases

* Tell between class members with/without self better

* TypeVar support

* Add bound/covariant

* Fix Enum reassignments

* Fix Random

* Fix import * over local declarations (Socket)

* Move interface

* Fix reference search

* Enable asyncio test

* More tests

* Enable dataclasses

* Add inspect and gzip

* More tests

* Add setting

* Add handling of import position relative to the variable

* Caching level changes

* Partial

* Rework

* Fix null

* Update baselines

* Functools pass

* Reverse stub merge

* Partial

* Partial ctypes

* Update variables post merge

* Fix ctypes

* Merge issues

* Fix typevar

* Add caching on construction

* Assorted fixes

* Named tuples

* Work around os stub oddness

* Fix generic qualified names

* Ignore lambdas

* Support named tuples as bases

* Baselines

* Test updates

* Named tuple tests

* PR feedback

* Using

* Move diagnostics to GenericBase

* PR feedback

* - Fix microsoft#1455: AF: Library module of type Stub has been analyzed already! (microsoft#1470)

- Restore task handling in unit tests
- Throw exception from Debug.Fail instead of FailFast

* Update baselines since __spec__ member was added

* Undo some of changes to match master.

* Fix null ref

* Resolve conflicts

* Fix null ref on bases due to new AnyStr behavior

* Handling dependencies in module persistence (microsoft#1471)

* Partial

* Models and views

* Restore log null checks

* Fix merge conflict

* Fix merge issue

* Null check

* Partial

* Partial

* Partial

* Fix test

* Partial

* Partial

* First test

* Baseline comparison

* Builtins

* Partial

* Type fixes

* Fix type names, part I

* Qualified name

* Properly write variables

* Partial

* Construct module from model

* Test

* Variable creations

* Factories

* Factories

* Split construction

* Restore

* Save builtins

* Test passes

* Qualified name

* Better export detection

* Test fixes

* More consistent qualified names

* Sys test

* Demo

* Complete sys write/read

* Partial

* Partial

* Test staility

* Perf bug

* Baseline, remove debug code, deactivate db

* Test fixes

* Test fix

* Simplify a bit

* Baselines and use : separator

* Baselines

* PR feedback

* Merge master

* Remove registry reference

* PR feedback

* PR feedback

* Restore persistence + update test

* Better handle persistent module in dependencies

* Undo

* Add location converter abstraction

* Store member location

* Fix merge issue

* Basic locations test

* Navigation

* Add test

* Update baselines

* Type restore - initial

* Remove debug code

* Partial

* Fix stub merge

* Various model fixes

* Improve module handling

* Qualified name improvements

* Fix unbound case

* Improve stub merge

* Fix qualified names of typing

* Handle stub-only modules

* Add tests for io, re and sys

* Better handle named tuple

* Handle module circular references

* Handle else in platform and version clauses + handle it in symbol collector

* Formatting

* Fix walk of multi-level if statement

* Unify package search in imports

* Fix tests

* Undo change

* Port changes from dbtype

* Partial

* Named tuple support

* Baseline updates

* Debug code

* Support types

* Properly compare class member declaring type

* Nested classes and functions persistence

* Undo debug

* Fix numpy restore

* Baselines

* Fix tests

* Update AnyStr test reflecting changes to AnyStr behavior

* Exclude baselines from git diff

* Fix gitattr

* Move git setting to root

* Try no path

* Test fixes

* Undo change

* Additional stub merge fixes

* Baseline updates

* Fix goto def

* Protect specific type creation

* Track documentaton source

* More reliable tests + simplification

* Typo

* Cleanup

* Basic classification

* Fix merge error

* Module unique id fixes

* Stricted check to save analysis

* Revert "Fix tests"

This reverts commit 247a8c3.

* Revert "Unify package search in imports"

This reverts commit 67fed10.

* Don't clear scope variables with inner classes

* Fix typo

* Many new tests

* Fix collections test

* Fix CTypes

* Initial

* Update test

* Merge issues

* Fix CTypes again

* Fix null bases

* Tell between class members with/without self better

* TypeVar support

* Add bound/covariant

* Fix Enum reassignments

* Fix Random

* Fix import * over local declarations (Socket)

* Move interface

* Fix reference search

* Enable asyncio test

* More tests

* Enable dataclasses

* Add inspect and gzip

* More tests

* Add setting

* Add handling of import position relative to the variable

* Caching level changes

* Partial

* Rework

* Fix null

* Update baselines

* Functools pass

* Reverse stub merge

* Partial

* Partial ctypes

* Update variables post merge

* Fix ctypes

* Merge issues

* Fix typevar

* Add caching on construction

* Assorted fixes

* Named tuples

* Work around os stub oddness

* Fix generic qualified names

* Ignore lambdas

* Support named tuples as bases

* Baselines

* Test updates

* Named tuple tests

* PR feedback

* Using

* Separate creation and population for better handling of forward references.

* Add dependency provider

* Merge issues

* Better class creation

* Partial

* Dependency providers

* Add IsPersistent flag

* Make special global scope

* Move dependency fetch earlier

* Replace AST walk by interface

* Make analysis sequence closer to master

* Undo some

* Undo debug

* Show dependencies in log

* null check

* Update tests

* Make sure factory looks in correct scope
Enable assertions in tests

* Separate stub dependencies

* Don't try and merge from empty stub

* Match generic parameters by name

* Use qualified names in generics
Minor tests cleanup

* Back to names

* Update tests

* usings

* Fix dependencies for stub

* Null check

* PR feedback

* PR feedback + fix merge issue in generics

* Test fixes + PR feedback

* Baseline updates

* PR feedback

* Restore original order

* Order

* Merge issues

* Make cancellation closer to original

* Pass cancellation to module walker

* Simplify

* Upgrade to 3.0

* Undo change + .NET 3

* Get back to 2.0

* Update components

* Display caching level
Round time to reasonble fractions.

* Fix null ref due to conflicting changes

* Null ref fix

* Simplify

* Pass AST whish is required during analysis when module does not have its own yet.

* tensorflow test

* Fix from import model
Add tensorflow test

* Undo debug

* Improve goto def in import and fix class base processing (microsoft#1529)

* Update signing project to 3.0

* Upgrade to .NET 3.0 (microsoft#1521)

* Upgrade to 3.0

* Get back to 2.0

* Add global.json for .NET SDK

* Ensure FileInfo is not relative in DirectoryInfoProxy (microsoft#1519)

* Ensure FileInfo is not relative in DirectoryInfoProxy

* Add tests

* Hadle import

* Update dependencies (microsoft#1526)

* Handle from import

* Add test

* Fix bad find and replace in AssertionsUtilities (microsoft#1527)

* Merge issues

* Add test (fails)

* using

* Fix merge of classes

* Fix regression in bases
Split merging into separate class

* Fix null refs

* Make common CanUpdateAnalysis check

* Usings

* Test fix

* Remove CompletionItemKind.None

* Add info to asserts

* PR feedback

* Undo test change, enable class reassignment
Fix stub merge for TypeVar

* Prevent merge of unrelated types

* more tests

* Fix variable update

* Null check

* Baselines

* Move analysis complete event

* More stable enumeration

* Improve restoring imports in tests

* Make extension for tests

* Add URI null check

* Fix completion typo

* Add assert on unresolved modules

* Add more details to assert

* Using

* To 3.7

* Implement check for big/little endian

* Sync

* Synchronization
Fix import model restore

* Add hard assert

* Fix exception

* Fix typo

* Add lock
  • Loading branch information
Mikhail Arkhipov authored Sep 16, 2019
1 parent 448c9cc commit 5280d34
Show file tree
Hide file tree
Showing 223 changed files with 8,043 additions and 1,988 deletions.
1 change: 1 addition & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
*.json -diff
4 changes: 2 additions & 2 deletions build/Common.Build.Core.settings
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,8 @@
</PropertyGroup>

<!-- This file is shared across desktop .NET and .NET Core builds.
Please do not put Windows-specific or dekstop-specific settings
here (these incluse C#, C++ or VSSDK settings and targets -->
Please do not put Windows-specific or desktop-specific settings
here (these include C#, C++ or VS SDK settings and targets -->
<PropertyGroup>
<Configuration Condition="'$(Configuration)' == ''">Debug</Configuration>
<VisualStudioVersion Condition="'$(VisualStudioVersion)' == ''">10.0</VisualStudioVersion>
Expand Down
7 changes: 7 additions & 0 deletions src/Analysis/Ast/Impl/Analyzer/Definitions/IAnalyzable.cs
Original file line number Diff line number Diff line change
Expand Up @@ -13,11 +13,18 @@
// See the Apache Version 2.0 License for specific language governing
// permissions and limitations under the License.

using Microsoft.Python.Analysis.Dependencies;

namespace Microsoft.Python.Analysis.Analyzer {
/// <summary>
/// Represents document that can be analyzed asynchronously.
/// </summary>
internal interface IAnalyzable {
/// <summary>
/// Returns object that can calculate dependencies of this entry.
/// </summary>
IDependencyProvider DependencyProvider { get; }

/// <summary>
/// Notifies document that analysis is about to begin.
/// </summary>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,6 @@
using System.Diagnostics;
using System.Linq;
using Microsoft.Python.Analysis.Documents;
using Microsoft.Python.Analysis.Extensions;
using Microsoft.Python.Analysis.Modules;
using Microsoft.Python.Analysis.Types;
using Microsoft.Python.Analysis.Values;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ public IMember GetValueFromList(ListExpression expression) {
var value = GetValueFromExpression(item) ?? UnknownType;
contents.Add(value);
}
return PythonCollectionType.CreateList(Module.Interpreter, contents, exact: expression.Items.Count <= MaxCollectionSize);
return PythonCollectionType.CreateList(Module, contents, exact: expression.Items.Count <= MaxCollectionSize);
}

public IMember GetValueFromDictionary(DictionaryExpression expression) {
Expand All @@ -72,7 +72,7 @@ public IMember GetValueFromDictionary(DictionaryExpression expression) {
var value = GetValueFromExpression(item.SliceStop) ?? UnknownType;
contents[key] = value;
}
return new PythonDictionary(Interpreter, contents, exact: expression.Items.Count <= MaxCollectionSize);
return new PythonDictionary(Module, contents, exact: expression.Items.Count <= MaxCollectionSize);
}

private IMember GetValueFromTuple(TupleExpression expression) {
Expand All @@ -81,7 +81,7 @@ private IMember GetValueFromTuple(TupleExpression expression) {
var value = GetValueFromExpression(item) ?? UnknownType;
contents.Add(value);
}
return PythonCollectionType.CreateTuple(Module.Interpreter, contents, exact: expression.Items.Count <= MaxCollectionSize);
return PythonCollectionType.CreateTuple(Module, contents, exact: expression.Items.Count <= MaxCollectionSize);
}

public IMember GetValueFromSet(SetExpression expression) {
Expand All @@ -90,7 +90,7 @@ public IMember GetValueFromSet(SetExpression expression) {
var value = GetValueFromExpression(item) ?? UnknownType;
contents.Add(value);
}
return PythonCollectionType.CreateSet(Interpreter, contents, exact: expression.Items.Count <= MaxCollectionSize);
return PythonCollectionType.CreateSet(Module, contents, exact: expression.Items.Count <= MaxCollectionSize);
}

public IMember GetValueFromGenerator(GeneratorExpression expression) {
Expand All @@ -110,14 +110,14 @@ public IMember GetValueFromComprehension(Comprehension node) {
switch (node) {
case ListComprehension lc:
var v1 = GetValueFromExpression(lc.Item) ?? UnknownType;
return PythonCollectionType.CreateList(Interpreter, new[] { v1 });
return PythonCollectionType.CreateList(Module, new[] { v1 });
case SetComprehension sc:
var v2 = GetValueFromExpression(sc.Item) ?? UnknownType;
return PythonCollectionType.CreateSet(Interpreter, new[] { v2 });
return PythonCollectionType.CreateSet(Module, new[] { v2 });
case DictionaryComprehension dc:
var k = GetValueFromExpression(dc.Key) ?? UnknownType;
var v = GetValueFromExpression(dc.Value) ?? UnknownType;
return new PythonDictionary(new PythonDictionaryType(Interpreter), new Dictionary<IMember, IMember> { { k, v } });
return new PythonDictionary(new PythonDictionaryType(Interpreter.ModuleResolution.BuiltinsModule), new Dictionary<IMember, IMember> { { k, v } });
}

return UnknownType;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@

namespace Microsoft.Python.Analysis.Analyzer.Evaluation {
internal sealed partial class ExpressionEval {
public IPythonInstance GetConstantFromLiteral(Expression expr, LookupOptions options) {
public IPythonInstance GetConstantFromLiteral(Expression expr) {
if (expr is ConstantExpression ce) {
switch (ce.Value) {
case string s:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -62,11 +62,12 @@ private IMember GetValueFromGeneric(IMember target, Expression expr) {
}

/// <summary>
/// Returns whether the arguments to Generic are valid
/// Determines if arguments to Generic are valid
/// </summary>
// TODO: move check to GenericClassBase. This requires extensive changes to SpecificTypeConstructor.
private bool GenericClassParameterValid(IReadOnlyList<IGenericTypeParameter> genericTypeArgs, IReadOnlyList<IMember> args, Expression expr) {
// All arguments to Generic must be type parameters
// e.g. Generic[T, str] throws a runtime error
// All arguments to Generic must be type parameters
// e.g. Generic[T, str] throws a runtime error
if (genericTypeArgs.Count != args.Count) {
ReportDiagnostics(Module.Uri, new DiagnosticsEntry(
Resources.GenericNotAllTypeParameters,
Expand All @@ -77,14 +78,14 @@ private bool GenericClassParameterValid(IReadOnlyList<IGenericTypeParameter> gen
return false;
}

// All arguments to Generic must be distinct
// All arguments to Generic must be distinct
if (genericTypeArgs.Distinct().Count() != genericTypeArgs.Count) {
ReportDiagnostics(Module.Uri, new DiagnosticsEntry(
Resources.GenericNotAllUnique,
GetLocation(expr).Span,
ErrorCodes.TypingGenericArguments,
Severity.Warning,
DiagnosticSource.Analysis));
Resources.GenericNotAllUnique,
GetLocation(expr).Span,
ErrorCodes.TypingGenericArguments,
Severity.Warning,
DiagnosticSource.Analysis));
return false;
}

Expand All @@ -97,24 +98,19 @@ private bool GenericClassParameterValid(IReadOnlyList<IGenericTypeParameter> gen
/// (if the former) on specific type (if the latter).
/// </summary>
private IMember CreateSpecificTypeFromIndex(IGenericType gt, IReadOnlyList<IMember> args, Expression expr) {
// TODO: move check to GenericClassBase. This requires extensive changes to SpecificTypeConstructor.
if (gt.Name.EqualsOrdinal("Generic")) {
var genericTypeArgs = args.OfType<IGenericTypeParameter>().ToArray();
if (!GenericClassParameterValid(genericTypeArgs, args, expr)) {
return UnknownType;
}

// Generic[T1, T2, ...] expression. Create generic base for the class.
return new GenericClassParameter(genericTypeArgs, Module);
}

// For other types just use supplied arguments
if (args.Count > 0) {
return gt.CreateSpecificType(new ArgumentSet(args, expr, this));
// Generic[T1, T2, ...] expression. Create generic base for the class.
return new GenericClassBase(genericTypeArgs, Module.Interpreter);
}

return UnknownType;
// For other types just use supplied arguments
return args.Count > 0 ? gt.CreateSpecificType(new ArgumentSet(args, expr, this)) : UnknownType;
}

private IReadOnlyList<IMember> EvaluateIndex(IndexExpression expr) {
var indices = new List<IMember>();
if (expr.Index is TupleExpression tex) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -105,9 +105,9 @@ private IMember GetValueFromBinaryOp(Expression expr) {

switch (leftTypeId) {
case BuiltinTypeId.List:
return PythonCollectionType.CreateConcatenatedList(Module.Interpreter, lc, rc);
return PythonCollectionType.CreateConcatenatedList(Module, lc, rc);
case BuiltinTypeId.Tuple:
return PythonCollectionType.CreateConcatenatedTuple(Module.Interpreter, lc, rc);
return PythonCollectionType.CreateConcatenatedTuple(Module, lc, rc);
}
}

Expand Down
32 changes: 18 additions & 14 deletions src/Analysis/Ast/Impl/Analyzer/Evaluation/ExpressionEval.cs
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@ public ExpressionEval(IServiceContainer services, IPythonModule module, PythonAs
public ModuleSymbolTable SymbolTable { get; } = new ModuleSymbolTable();
public IPythonType UnknownType => Interpreter.UnknownType;
public Location DefaultLocation { get; }
public IPythonModule BuiltinsModule => Interpreter.ModuleResolution.BuiltinsModule;

public LocationInfo GetLocationInfo(Node node) => node?.GetLocation(this) ?? LocationInfo.Empty;

Expand Down Expand Up @@ -188,7 +189,7 @@ public IMember GetValueFromExpression(Expression expr, LookupOptions options = L
m = null;
break;
default:
m = GetValueFromBinaryOp(expr) ?? GetConstantFromLiteral(expr, options);
m = GetValueFromBinaryOp(expr) ?? GetConstantFromLiteral(expr);
break;
}
if (m == null) {
Expand Down Expand Up @@ -236,29 +237,32 @@ private IMember GetValueFromMember(MemberExpression expr) {
return null;
}

IPythonInstance instance = null;
var m = GetValueFromExpression(expr.Target);
if (m is IPythonType typeInfo) {
var member = typeInfo.GetMember(expr.Name);
// If container is class/type info rather than the instance, then the method is an unbound function.
// Example: C.f where f is a method of C. Compare to C().f where f is bound to the instance of C.
if (member is PythonFunctionType f && !f.IsStatic && !f.IsClassMethod) {
f.AddReference(GetLocationOfName(expr));
return f.ToUnbound();
}
instance = new PythonInstance(typeInfo);
if (m == null) {
return UnknownType;
}

instance = instance ?? m as IPythonInstance;
var type = m?.GetPythonType(); // Try inner type

var type = m.GetPythonType();
var value = type?.GetMember(expr.Name);
type?.AddMemberReference(expr.Name, this, GetLocationOfName(expr));

if (type is IPythonModule) {
return value;
}

IPythonInstance instance = null;
if (m == type) {
// If container is class/type info rather than the instance, then the method is an unbound function.
// Example: C.f where f is a method of C. Compare to C().f where f is bound to the instance of C.
if (value is PythonFunctionType f && f.DeclaringType != null && !f.IsStatic && !f.IsClassMethod) {
f.AddReference(GetLocationOfName(expr));
return f.ToUnbound();
}
instance = new PythonInstance(type);
}

instance = instance ?? m as IPythonInstance;

// Class type GetMember returns a type. However, class members are
// mostly instances (consider self.x = 1, x is an instance of int).
// However, it is indeed possible to have them as types, like in
Expand Down
6 changes: 3 additions & 3 deletions src/Analysis/Ast/Impl/Analyzer/Handlers/AssignmentHandler.cs
Original file line number Diff line number Diff line change
Expand Up @@ -58,14 +58,14 @@ public void HandleAssignment(AssignmentStatement node) {
}

foreach (var ne in node.Left.OfType<NameExpression>()) {
IScope scope;
if (Eval.CurrentScope.NonLocals[ne.Name] != null) {
Eval.LookupNameInScopes(ne.Name, out var scope, LookupOptions.Nonlocal);
Eval.LookupNameInScopes(ne.Name, out scope, LookupOptions.Nonlocal);
scope?.Variables[ne.Name].Assign(value, Eval.GetLocationOfName(ne));
continue;
}

if (Eval.CurrentScope.Globals[ne.Name] != null) {
Eval.LookupNameInScopes(ne.Name, out var scope, LookupOptions.Global);
Eval.LookupNameInScopes(ne.Name, out scope, LookupOptions.Global);
scope?.Variables[ne.Name].Assign(value, Eval.GetLocationOfName(ne));
continue;
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,6 @@ public ConditionalHandler(AnalysisWalker walker) : base(walker) {
}

public bool HandleIf(IfStatement node)
=> node.WalkIfWithSystemConditions(Walker, Ast.LanguageVersion, _platformService.IsWindows);
=> node.WalkIfWithSystemConditions(Walker, Ast.LanguageVersion, _platformService);
}
}
22 changes: 12 additions & 10 deletions src/Analysis/Ast/Impl/Analyzer/Handlers/FromImportHandler.cs
Original file line number Diff line number Diff line change
Expand Up @@ -68,11 +68,13 @@ private void AssignVariables(FromImportStatement node, IImportSearchResult impor
if (!string.IsNullOrEmpty(memberName)) {
var nameExpression = asNames[i] ?? names[i];
var variableName = nameExpression?.Name ?? memberName;
var variable = variableModule.Analysis?.GlobalScope?.Variables[memberName];
var exported = variable ?? variableModule.GetMember(memberName);
var value = exported ?? GetValueFromImports(variableModule, imports as IImportChildrenSource, memberName);
// Do not allow imported variables to override local declarations
Eval.DeclareVariable(variableName, value, VariableSource.Import, nameExpression, CanOverwriteVariable(variableName, node.StartIndex));
if (!string.IsNullOrEmpty(variableName)) {
var variable = variableModule.Analysis?.GlobalScope?.Variables[memberName];
var exported = variable ?? variableModule.GetMember(memberName);
var value = exported ?? GetValueFromImports(variableModule, imports as IImportChildrenSource, memberName);
// Do not allow imported variables to override local declarations
Eval.DeclareVariable(variableName, value, VariableSource.Import, nameExpression, CanOverwriteVariable(variableName, node.StartIndex));
}
}
}
}
Expand All @@ -84,7 +86,7 @@ private void HandleModuleImportStar(PythonVariableModule variableModule, bool is
}

// If __all__ is present, take it, otherwise declare all members from the module that do not begin with an underscore.
var memberNames = isImplicitPackage
var memberNames = isImplicitPackage
? variableModule.GetMemberNames()
: variableModule.Analysis.StarImportMemberNames ?? variableModule.GetMemberNames().Where(s => !s.StartsWithOrdinal("_"));

Expand All @@ -109,7 +111,7 @@ private void HandleModuleImportStar(PythonVariableModule variableModule, bool is

private bool CanOverwriteVariable(string name, int importPosition) {
var v = Eval.CurrentScope.Variables[name];
if(v == null) {
if (v == null) {
return true; // Variable does not exist
}
// Allow overwrite if import is below the variable. Consider
Expand All @@ -118,10 +120,10 @@ private bool CanOverwriteVariable(string name, int importPosition) {
// from A import * # brings another x
// x = 3
var references = v.References.Where(r => r.DocumentUri == Module.Uri).ToArray();
if(references.Length == 0) {
if (references.Length == 0) {
// No references to the variable in this file - the variable
// is imported from another module. OK to overwrite.
return true;
return true;
}
var firstAssignmentPosition = references.Min(r => r.Span.ToIndexSpan(Ast).Start);
return firstAssignmentPosition < importPosition;
Expand Down Expand Up @@ -151,7 +153,7 @@ private void SpecializeFuture(FromImportStatement node) {
var printNameExpression = node.Names.FirstOrDefault(n => n?.Name == "print_function");
if (printNameExpression != null) {
var fn = new PythonFunctionType("print", new Location(Module), null, string.Empty);
var o = new PythonFunctionOverload(fn.Name, new Location(Module));
var o = new PythonFunctionOverload(fn, new Location(Module));
var parameters = new List<ParameterInfo> {
new ParameterInfo("*values", Interpreter.GetBuiltinType(BuiltinTypeId.Object), ParameterKind.List, null),
new ParameterInfo("sep", Interpreter.GetBuiltinType(BuiltinTypeId.Str), ParameterKind.KeywordOnly, null),
Expand Down
2 changes: 1 addition & 1 deletion src/Analysis/Ast/Impl/Analyzer/Handlers/ImportHandler.cs
Original file line number Diff line number Diff line change
Expand Up @@ -180,7 +180,7 @@ private PythonVariableModule GetOrCreateVariableModule(in string fullName, in Py
return variableModule;
}

variableModule = new PythonVariableModule(fullName, Interpreter);
variableModule = new PythonVariableModule(fullName, Eval.Interpreter);
_variableModules[fullName] = variableModule;
parentModule?.AddChildModule(memberName, variableModule);
return variableModule;
Expand Down
3 changes: 0 additions & 3 deletions src/Analysis/Ast/Impl/Analyzer/LibraryAnalysis.cs
Original file line number Diff line number Diff line change
Expand Up @@ -15,12 +15,9 @@

using System.Collections.Generic;
using System.Linq;
using Microsoft.Python.Analysis.Analyzer.Evaluation;
using Microsoft.Python.Analysis.Diagnostics;
using Microsoft.Python.Analysis.Documents;
using Microsoft.Python.Analysis.Modules;
using Microsoft.Python.Analysis.Values;
using Microsoft.Python.Core;
using Microsoft.Python.Core.Diagnostics;
using Microsoft.Python.Parsing.Ast;

Expand Down
Loading

0 comments on commit 5280d34

Please sign in to comment.