V1.5 to Master

[mayanje]

V1.5

This release 1.5 mainly implements the construction of Control Flow Graph (CFG) for both Cobol and TypeCobol programs, and the support of Data Flow Analysis (DFA) for Cobol programs only.

This version also provide support for a new quality step and external code analyzer loading (DLL).

A lot of bug have been fixed.

https://en.wikipedia.org/wiki/Control-flow_graph
https://en.wikipedia.org/wiki/Data-flow_analysis

What's New

Semantic Domain (#1262, #1264)

• New Semantic Domain Objects and Symbol table for better Symbols and Types representation (Register, represent and implement semantic domain objects (A01.T01) #1262, Integrate SemanticDomain #1721, Implement real Symbol Tables #1278)
• Store Section and Paragraph in the new Symbol table (Elaborate Section and Paragraph in the new SymbolTable. #1394)
• Integrate SemanticDomain (Integrate SemanticDomain #1721, 1721 semantic domain lite #1722)
• Add Listener/Dispatcher interfaces for node creation/parsing (Close #1263 Add Listener/Dispatcher interfaces fornode creation/parsing action #1597).

Control Flow Graph (#1260, #1396)

• Implement the data structure of a CFG graph (Implement the data structure of the CFG graph. (A03.T01) #1265)
• Integrate all TypeCobol control flow instructions (List all TypeCobol control instructions (A04.T01) #1266).
• Build CFG graph for a TypeCobol program (Build CFG Graphs from a TypeCobol Program (A04.T02) #1267)
• Generate .dot file for a CFG program (Generate the .dot file of a CFG graph. (A05.T01) #1268)
• Cobol specificities
  • Handle Program CFG without PROCEDURE-DIVISION (CFG/DFA Check behaviour with no PROCEDURE DIVISION #1502)
  • Ignore EXEC-SQL Statement Out of a PROCEDURE DIVISION (CFG Ignore EXEC SQL Statement Out of PROCEDURE DIVISION #1501)
• Tests CFG .dot graph and visualiation (Test CFG dot graphs and visualization. (A05.T02) #1269)

Data Flow Analysis (#1260, #627, #1720, #1750)

• Increase the CFG graph with data structures for DFA (Increase the CFG graph with data structures and fields for DFA. (A06.T01). #1270)
• Identify Cobol Instruction that target DFA (Identify TypeCobol instructions targeted by DFA. (A07.T01) #1271)
• Apply DFA algorithms (Apply the DFA algorithms. (A07.T02) #1272).
• Generate the .dot file of a DFG (Generate the .dot file of a DFG graph. (A08.T01) #1273)
• Test DFG .dot graphs and visualization (Test DFG dot graphs and visualization. (A08.T02) #1274)
• Diagnostics
  • Dataflow diagnostics and TypeCobol diagnostics unification (Unify DataFlow diagnostics and TypeCobol diagnostics. #1285)
• Lite DFA Integration for Cobol only (Integrate DFA in CFG_SemanticDomainLite #1750)
• Add DataDefinition nodes as CFG instructions in the START block of a DFG (Integrate DFA in CFG_SemanticDomainLite #1750)

CFG/DFA in practice

• Quality rule: Identify module called by a zcallpgm (Identify the modules called by a CALL statement. (A09.T01) #1275)

Cobol Parser improvments

• Better handling of scan state dependent tokens(Tokens replaced need the MultilineScanState to determine their type #428)
• Forbid SYNC clause with Strict Type (Forbird SYNC clause with Strict Type #960, WI 969 Forbid Sync Clause for Strict Types #1729)
• Handle SYNC LEFT and RIGHT (Handle SYNC LEFT and RIGHT #1200)

Semantic Domain towards Analyzers for Quality rules

• cfg semantic domain lite introduce analyzers (1260 CFG introduce analyzers #1739)

Performance, tests and Stats (issues and improvements)

This version decrease performance.
Full parsing of a source can be 5% to 20% slower than 1.4.3
CFG/DFA activation doesn't impact performance so much (0% to 15%)

Incremental mode is 42% to 112% slower.
CFG/DFA activation in incremental doesn't impact performance so much (0% to 15%)

• Update performance tests to take into account CFG/DFA construction (Add performance tests in in v1.5 #1863, Add performance tests in v1.5 to use CFG/DFA options #1864, Update Typecobol performance tests #1851)
• Cobol Parser
  • Use hand-code algorithm instead of regular expression, to parse PICTURE strings (Optimize Picture String Validator #1357)
• Update GlobalStorage test with new Semantic errors (Diagnostics set on FunctionDeclaration should be more precise on specify which Token, SymbolReference, ... is in error #1133)

TypeCobol Options

• Add a new option to specify the path of the log file TypeCobol.CLI.Log (Add an option to specify the path of the log file TypeCobol.CLI.log #1513)

Towards Quality Rules

• Add Rules and Violations concepts to the parser (Add Rules and Violations concepts to the parser #1837)
• Support external analyzers (Support external analyzers #1825)
• Introduce new QualityCheck compilation step (IAnalyser and new execution step for quality rules #1816)
• Add .xmldiag format for error output in CLI (New diagnostics output format for CLI #1845)
• Remove IASTAnalyzer, use QualityAnalyzer concept instead (Add mechanism for external analyzers to check compiler-directing statements #1869)

Minor and other changes

• Change Internal class visibility and scope (Make class SetStatementForConditions public #1766)
• Internal Code Refactoring
  • Remove unused dependencies: (Razor, Moq...) (Remove unused dependencies #1679, 1679 remove unused dependencies #1682)
  • Disable CodeAnalysis for Debug-like configurations (Improve compilation time #1800)
  • Align all csproj configuration and properties to use the common ones (Common.props reintegration of multiple .csproj paramaters #1627)
  • Upgrade NLog to v4.7.2 (Upgrade to NLog 4.7.2 #1772)

Bug Fix

• Language Server
  • LSP Signature Help Request raise exception (LSP Signature Help Request on the first line raise exception #1811)
  • Send notification for all logged exceptions (Send notification for all logged exceptions #1622)
• Cobol EI Parser
  • Exception on sources with broken format (Exception on sources with broken format #1852)
  • Tokens replaced need the MultilineScanState to determine their type (Tokens replaced need the MultilineScanState to determine their type #428)
  • Complete the missing copies list with all levels of inclusion (Complete the missing copies list with all levels of inclusion #1808)
  • Wrong inital values for data using continuation lines (Wrong inital values for data using continuation lines #1791)
  • Bug in tag/placeholder replacing (Bug in tag/placeholder replacing #1785)
  • Continuation line parsing breaks when mixed with comment lines (Continuation line parsing breaks when mixed with comment lines #1777)
  • Copy replacing for a level(Copy replacing for a level #1753)
  • Strict parsing of *CBL directives(Strict parsing of *CBL directives  #1748)
  • Invalid cast in CrossCheck (Invalid cast in CrossCheck #1746)
  • Empty IF ELSE should be allowed in grammar but checked in CrossChecker (Empty IF ELSE should be allowed in grammar but checked in CrossChecker #1676)
  • Bug in DataDefinition with continuation lines (Bug in DataDefinition with continuation lines #1671)
  • DataRenames node are generated as children of previous node (DataRenames node are generated as children of previous node #1192)
  • Unterminated literals doesn't generate an error (Unterminated literals doesn't generate an error #1160)
  • TALLY is changed to ALL in error message (TALLY is changed to ALL in error message #1120)
  • Replaced Token doesn't work inside DataDivision (Replaced Token doesn't work inside DataDivision #599)
  • Arguments of standard call are not controlled (Arguments of standard call are not controlled #532)
  • Continuation line (Continuation line #457)
  • Tokens replaced need the MultilineScanState to determine their type (Tokens replaced need the MultilineScanState to determine their type #428)
  • EXEC SQL INCLUDE on multiple lines doesn't works (EXEC SQL INCLUDE on multiple lines doesn't works #321)
  • Check duplicate paragraph names with semantic data owner (Compile error on programs with 2 paragraphs with the same name on procedure division #1710)
  • Replace all placeholders in partial word replace operations (Incorrect word replacing when placeholder appears multiple times #1780)
  • Fix numeric literal detection in scanner after THROUGH/THRU (Range value in condition levels are not parsed correctly #1778)
  • Allow "VALUE ARE"/"VALUES IS" in data condition value clauses (Allow "VALUE ARE" and "VALUES IS" syntaxes #1782)
  • Allow modification on input indices in procedures (Error on input table index SET statement #1789)
  • handle copy replacing with 4colon ol (Handle copy replacing with ==::== #1752)

[smedilol]

Can you add the whole changelog of v1.5 in this PR ?
This PR also target to validate this changelog.

[mayanje]

Sure I am working on this aspect, I will inform you when it will be ready

[mayanje]

In the change log only issues of Milestone v1.5 have been taken in account.

[mayanje]

Je vais aussi intégrer les issues closed des milestones CobolEditorEI et Quality Rule

[smedilol]

Everything about new SymbolTable can be removed as it was abandonned (like Global-storage visibility integration in the new Symbol table (#1391, #805)).
You can also removed references to PR. We only need references to issues.

[fm-117]

Updated changelog:

• fix typo
• move quality analyzer issues references to proper section (instead of bug fixes)

[mayanje]

Change Log shall be ok now

[fm-117]

• removed codegen from changelog as simplified codegen has not been merged, it was only a test
• fixed issue reference for QualityCheck
• moved Improve compilation time #1800 reference to proper section
• removed Filler and variable without name must be added to SymbolTable #216 as it has been cancelled rather than closed

You can check my edits of the changelog.

[smedilol]

I've made a small update:

• I removed PR references
• I removed one line about new Symbol table

You can check my edits of the changelog.

[smedilol]

Other update:

• regroup bug fix by category
• remove some duplicate entries
• rename TypeCobol Options to Command Line Interface

You can check my edits of the changelog.

@mayanje Can you post figures about performance of v1.5 VS v1.4.3 ? Also with differents options without CFG/DFA, with CFG standard, CFG extented, DFA.

[mayanje]

@smedilol what do you mean by posting figures?
There are four différents tests v1.4.3, v1.5 NoCfg , v1.5CfgStandard and v1.5DFAExtended, results are cvs files.

[smedilol]

@mayanje I mean the result of these tests.
The goal is to know if performance have improved or decreased.

[smedilol]

Another update:
Add more info about performance.
Add 2 info in the summary.