Code

Elastic6 work in progress#2373

Code

Merged

tonytw1 merged 194 commits intomasterguardian/grid:masterfrom elastic6guardian/grid:elastic6Copy head branch name to clipboardJan 29, 2019

+4,041-556

Commits

Commits on Dec 6, 2018

• Rename create index method to make it explicit that this call only makes a specific type of index (one with image mappings).

  

  Tony McCrae

  committed
  7b7976d

  Copy full SHA for 7b7976d

• Remove duplicate suggestion mapper.

  

  Tony McCrae

  committed
  a1bb477

  Copy full SHA for a1bb477

Commits on Dec 7, 2018

• Remove unused identifier settings function.

  

  Tony McCrae

  committed
  83bb510

  Copy full SHA for 83bb510

• Initial port of image index defination to elastic4s DSL.

  

  Tony McCrae

  committed
  9eb3b08

  Copy full SHA for 9eb3b08

• Fixes incorrectly migrated mappings and index settings.

  

  Tony McCrae

  committed
  d0d73e9

  Copy full SHA for d0d73e9

• Introduce an interface to document the elastic search operations which need to be supported.

  Show description for 8202156
  

  Tony McCrae

  committed
  8202156

  Copy full SHA for 8202156

Commits on Dec 11, 2018

• Add a minimal Elastic search 6 test instance on port 9206.

  Show description for cc57305
  

  Tony McCrae

  committed
  cc57305

  Copy full SHA for cc57305

• Spike. Thrall Syndication ops tests pass with against Elastic 1.7 and 6.4.

  Show description for 44a6569
  

  Tony McCrae

  committed
  44a6569

  Copy full SHA for 44a6569

• Refactor; rename.

  Show description for 34e0469
  

  Tony McCrae

  committed
  34e0469

  Copy full SHA for 34e0469

• Level up the to elastic4s 6.4.

  Show description for d573705
  

  Tony McCrae

  committed
  d573705

  Copy full SHA for d573705

• Clean up; deduplicate tests.

  

  Tony McCrae

  committed
  4f0674c

  Copy full SHA for 4f0674c

• Moving towards the restoration of the .executeAndLog functionality in the version 6 client.

  

  Tony McCrae

  committed
  a416836

  Copy full SHA for a416836

• elastic4s generally returns happy failures; remap this into failed futures for consistancy with existing driver.

  

  Tony McCrae

  committed
  edebdc5

  Copy full SHA for edebdc5

• Catch no match.

  

  Tony McCrae

  committed
  5361a2b

  Copy full SHA for 5361a2b

Commits on Dec 12, 2018

• Thrall ElasticClient returns native elastic search classes which are probably not needed by the callers.

  Show description for 24ea602
  

  Tony McCrae

  committed
  24ea602

  Copy full SHA for 24ea602

• Produce a local dev elastic6 on port 9206.

  

  Tony McCrae

  committed
  4a42e4c

  Copy full SHA for 4a42e4c

• Take advantage of the ElasticSearchVersion interface to insert a write to both versions facade infront of the Thrall consumer.

  Show description for 41b5673
  

  Tony McCrae

  committed
  41b5673

  Copy full SHA for 41b5673

• Introduce some testing around specific elastic search 6 operations.

  Show description for c4bef32
  

  Tony McCrae

  committed
  c4bef32

  Copy full SHA for c4bef32

Commits on Dec 17, 2018

• Elastic6 implementation of update user meta data.

  Show description for ea1364f
  

  Tony McCrae

  committed
  ea1364f

  Copy full SHA for ea1364f

• Clean up; with tests in place we can now safely reduce this boolean expression.

  

  Tony McCrae

  committed
  2d7b314

  Copy full SHA for 2d7b314

• Restoring refresh usages rights scripts.

  

  Tony McCrae

  committed
  5de2048

  Copy full SHA for 5de2048

• Port photoshoot suggestion script.

  

  Tony McCrae

  committed
  1e3628f

  Copy full SHA for 1e3628f

• Coverage for update syndication rights.

  

  Tony McCrae

  committed
  bf54624

  Copy full SHA for bf54624

Commits on Dec 18, 2018

• Test robustness; reinstate the wait for green cluster check.

  Show description for 8f73698
  

  Tony McCrae

  committed
  8f73698

  Copy full SHA for 8f73698

• Ensure indexes before elastic search test.

  

  Tony McCrae

  committed
  a1efe9d

  Copy full SHA for a1efe9d

• Elastic6 index needs to be single shade, no replicas in dev/test.

  Show description for a82b4f9
  

  Tony McCrae

  committed
  a82b4f9

  Copy full SHA for a82b4f9

• Increase timeout for deployment of ES1 and ES6 docker images.

  

  Tony McCrae

  committed
  0e74709

  Copy full SHA for 0e74709

• Config for ES6 hostname.

  

  Tony McCrae

  committed
  d0ae0c3

  Copy full SHA for d0ae0c3

• Clean up; push ES6 index shard and replicas count up to config.

  

  Tony McCrae

  committed
  0f461e3

  Copy full SHA for 0f461e3

• Elastic6 delete image implementation.

  Show description for dc09668
  

  Tony McCrae

  committed
  dc09668

  Copy full SHA for dc09668

• Unroll deprecated count document type argument.

  Show description for 5cb3c7e
  

  Tony McCrae

  committed
  5cb3c7e

  Copy full SHA for 5cb3c7e

• Clean up; using count API for delete lookup seems to remove the need to the indexing wait.

  

  Tony McCrae

  committed
  5d6f128

  Copy full SHA for 5d6f128

• Delete image usages implementation.

  

  Tony McCrae

  committed
  3e86603

  Copy full SHA for 3e86603

• Clean up; push test data towards Fixtures.

  

  Tony McCrae

  committed
  0797f2d

  Copy full SHA for 0797f2d

Commits on Dec 19, 2018

• Update image usages elastic6 implementation.

  

  Tony McCrae

  committed
  9fe885b

  Copy full SHA for 9fe885b

• Delete syndication rights elastic 6 implementation.

  

  Tony McCrae

  committed
  6dad14d

  Copy full SHA for 6dad14d

• Elastic6 implementations for add and remove leases.

  

  Tony McCrae

  committed
  a0f35cf

  Copy full SHA for a0f35cf

• Test waits.

  

  Tony McCrae

  committed
  a0f0a13

  Copy full SHA for a0f0a13

• Elastic6 implementation of replace leases.

  

  Tony McCrae

  committed
  ee26759

  Copy full SHA for ee26759

• Elastic6 implementation of set collections.

  

  Tony McCrae

  committed
  c7e7eea

  Copy full SHA for c7e7eea

• Elastic6 implementation for add and delete exports.

  

  Tony McCrae

  committed
  3a2f1fc

  Copy full SHA for 3a2f1fc

• Test wait; pushing into one place ahead of optimisation.

  

  Tony McCrae

  committed
  3f307b9

  Copy full SHA for 3f307b9

• Revert commit of local changes

  

  Tony McCrae

  committed
  fca6e79

  Copy full SHA for fca6e79

• Merge branch 'master' into elastic6

  tonytw1

  authored
  9511044

  Copy full SHA for 9511044

• Apply elastic search method tests to Elastic 1 and 6 implementations.

  Show description for bbe3282
  

  Tony McCrae

  committed
  bbe3282

  Copy full SHA for bbe3282

• Extracting an interface to document the elastic search operations which need to be ported to elastic 6.

  Show description for e1d5bfb
  

  Tony McCrae

  committed
  e1d5bfb

  Copy full SHA for e1d5bfb

• Push down private method.

  

  Tony McCrae

  committed
  9e28378

  Copy full SHA for 9e28378

• Refactor; move rendering of aggregation responses from elastic to controllers package.

  Show description for be7bd6d
  

  Tony McCrae

  committed
  be7bd6d

  Copy full SHA for be7bd6d

• Refactor; collecting the model classes needed to talk to Elastic search into a model grouping.

  Show description for 4a5e79b
  

  Tony McCrae

  committed
  4a5e79b

  Copy full SHA for 4a5e79b

Commits on Dec 21, 2018

• Refactor; elastic search interface returns images as case classes rather than JsValues

  

  Tony McCrae

  committed
  e0a970b

  Copy full SHA for e0a970b

• Work in progress; setting up for Elastic6 implmentation of the media-api elasticsearch interface

  

  Tony McCrae

  committed
  d9ed584

  Copy full SHA for d9ed584

Commits on Jan 2, 2019

• Clean up; consistent mapping style.

  

  Tony McCrae

  committed
  a2fb5b0

  Copy full SHA for a2fb5b0

• Remove unused deleteIndex method.

  Show description for 098ab55
  

  Tony McCrae

  committed
  098ab55

  Copy full SHA for 098ab55

• Logging; prefix elastic6 operations with ES6.

  

  Tony McCrae

  committed
  d34774a

  Copy full SHA for d34774a

• Additonal documenting tests for existing indexImage behaviour.

  

  Tony McCrae

  committed
  28c124d

  Copy full SHA for 28c124d

• Reinstate indexing wait.

  

  Tony McCrae

  committed
  2f19f49

  Copy full SHA for 2f19f49

• Marking TODO for missing redaction of reindex requests test coverage.

  

  Tony McCrae

  committed
  879a2e8

  Copy full SHA for 879a2e8

• Extract reindex redaction which Elastic1 applies to non initial index requests and apply equally to Elastic6.

  Show description for 4e08828
  

  Tony McCrae

  committed
  4e08828

  Copy full SHA for 4e08828

• Pick up elastic6 hostname from EC2 tags in non dev environments.

  

  Tony McCrae

  committed
  f29539c

  Copy full SHA for f29539c

• No defaults for non dev elastic6 replica and shards; these are important enough to force them to be set.

  

  Tony McCrae

  committed
  b687822

  Copy full SHA for b687822

• Imports.

  

  Tony McCrae

  committed
  603df6b

  Copy full SHA for 603df6b

• Document our updated understanding of how getCurrentIndices is used.

  

  Tony McCrae

  committed
  ef3f583

  Copy full SHA for ef3f583

Commits on Jan 3, 2019

• Setting up for optional dual running of ES1 and ES6.

  Show description for 680e028
  

  Tony McCrae

  committed
  680e028

  Copy full SHA for 680e028

• Refactor; split out actor health check which was embedded in the elastic health check; compose response to contain all current problems.

  Show description for 865c861
  

  Tony McCrae

  committed
  865c861

  Copy full SHA for 865c861

• Move elastic search health check specifics to ES1 and ES6 implementations.

  Show description for 03cc32c
  

  Tony McCrae

  committed
  03cc32c

  Copy full SHA for 03cc32c

• Active ES1 and ES6 implementations can be toggled by the presence of their config. Allows deployment of new code against ES1 index only.

  

  Tony McCrae

  committed
  336a9b9

  Copy full SHA for 336a9b9

• Test config; no replicas in test.

  

  Tony McCrae

  committed
  ffe4057

  Copy full SHA for ffe4057

• Deployment enters an infinite startup loop if no Elastic6 is available; revert to named host.

  

  Tony McCrae

  committed
  ddda11a

  Copy full SHA for ddda11a

Commits on Jan 4, 2019

• Refactor - push EC2 host discovery into seperate package; not an elastic concern even if elastic is the only user.

  

  Tony McCrae

  committed
  4db7cb9

  Copy full SHA for 4db7cb9

• Level up to 6.5.4 latest elastic release.

  

  Tony McCrae

  committed
  47a3c2a

  Copy full SHA for 47a3c2a

• Elastic single node discovery flag for 6.5 in dev and test.

  

  Tony McCrae

  committed
  1e05bba

  Copy full SHA for 1e05bba

• Merge branch 'elastic6' into elastic6-media-api

  

  Tony McCrae

  committed
  2675991

  Copy full SHA for 2675991

• Refactor - collecting Elasticsearch 1 implementation specific classes into a package.

  

  Tony McCrae

  committed
  b9c8f6d

  Copy full SHA for b9c8f6d

• Restor; push the init' of the elastic 1 search filters class down into the Elastic1 constructor.

  

  Tony McCrae

  committed
  089e8e0

  Copy full SHA for 089e8e0

• Marking private method to reduce dev cognitive load.

  Show description for 27e4f75
  

  Tony McCrae

  committed
  27e4f75

  Copy full SHA for 27e4f75

• WIP Begin porting filters DSL to elastic4s. Rename tests to avoid name clash with thrall in IDE.

  Show description for 57d7035
  

  Tony McCrae

  committed
  57d7035

  Copy full SHA for 57d7035

• WIP Navive porting of SearchFilters; filters DSL continues to in insulate us the the underlying elastic version.

  

  Tony McCrae

  committed
  168c326

  Copy full SHA for 168c326

• WIP Restore unresolved WIP

  

  Tony McCrae

  committed
  0b56363

  Copy full SHA for 0b56363

Commits on Jan 7, 2019

• Refactor - isolate non DSL native call to elastic mustNot method by introducing a specific filters DSL method for the optimized must with mustNot query.

  Show description for 8ec374c
  

  Tony McCrae

  committed
  8ec374c

  Copy full SHA for 8ec374c

• Naive port of filters.mustWithMustNot to Elastic6.

  

  Tony McCrae

  committed
  e1cb666

  Copy full SHA for e1cb666

• Testing. Correctly implementation of delete test images and delete all before tests for a more predictable developer experiences when working in an IDE (which may not always call afterAll).

  

  Tony McCrae

  committed
  b67c8bd

  Copy full SHA for b67c8bd

• We'd like to use type safe Images if possible. Some of the existing test images are not happy to be round tripped through Elastic6 and this test documents this.

  Show description for 21e93b3
  

  Tony McCrae

  committed
  21e93b3

  Copy full SHA for 21e93b3

• Test images loading correctly from Elastic6.

  

  Tony McCrae

  committed
  6749872

  Copy full SHA for 6749872

• Clean up.

  

  Tony McCrae

  committed
  ff62370

  Copy full SHA for ff62370

• Optimise test setup waits with eventually clauses.

  Show description for 97cbbb1
  

  Tony McCrae

  committed
  97cbbb1

  Copy full SHA for 97cbbb1

• Optimise test waits. 50s -> ~ 1s

  Show description for 18ccb7c
  

  Tony McCrae

  committed
  18ccb7c

  Copy full SHA for 18ccb7c

• Log elasticsearch hits which cannot be resolved to Images.

  

  Tony McCrae

  committed
  bdbb6f7

  Copy full SHA for bdbb6f7

• WIP We only have 1 failing test with which to assess ES1 vs ES6.

  

  Tony McCrae

  committed
  8af03e5

  Copy full SHA for 8af03e5

• Fixes lease end date queries; ES1 date filter DSL relied on ES1 Builder having mutable state. ES6 client doesn't behave the same way but the filter DSL looks very functional so is likely safe.

  Show description for bb15a7f
  

  Tony McCrae

  committed
  bb15a7f

  Copy full SHA for bb15a7f

Commits on Jan 8, 2019

• Setting up for implementation of QueryBuilder.

  

  Tony McCrae

  committed
  d7b137a

  Copy full SHA for d7b137a

• Clean up; resolve not method clash between filter and Elastic DSLs.

  

  Tony McCrae

  committed
  8b36031

  Copy full SHA for 8b36031

• Refactor; move tests to match implementation package structure.

  

  Tony McCrae

  committed
  471d5a9

  Copy full SHA for 471d5a9

• Refactor; rename fixtures class.

  

  Tony McCrae

  committed
  2b828bc

  Copy full SHA for 2b828bc

• Correct test package name.

  

  Tony McCrae

  committed
  96f8ef2

  Copy full SHA for 96f8ef2

• Back filling QueryBuilder test coverage.

  

  Tony McCrae

  committed
  f77248d

  Copy full SHA for f77248d

• Elastic6 QueryBuilder should return a Query equivalent to the Elastic1 implementation. Force the Elastic6 service to accomodate this so that the diffs between the 2 query builder implementations ar…

  Show description for d5e2b26
  

  Tony McCrae

  committed
  d5e2b26

  Copy full SHA for d5e2b26

• Combine query builder and filter inputs in a manner similar to the ES1 implementation.

  

  Tony McCrae

  committed
  f477d98

  Copy full SHA for f477d98

• Helper to render ES6 search request body JSON in tests; use as a sanity check to visually cross check that ES1 and ES6 are sending vaguely simalar requests over the wire.

  

  Tony McCrae

  committed
  2122c61

  Copy full SHA for 2122c61

• Spike; prove that we can test drive out the ES6 Query builder behaviour by observing the ES1 behaviour and then porting the tests.

  

  Tony McCrae

  committed
  3b198ae

  Copy full SHA for 3b198ae

• Port query builder negation behaviour.

  

  Tony McCrae

  committed
  a6b73d9

  Copy full SHA for a6b73d9

• Port word field match.

  

  Tony McCrae

  committed
  d26ac97

  Copy full SHA for d26ac97

• Port date range clause.

  

  Tony McCrae

  committed
  0b1598b

  Copy full SHA for 0b1598b

• Port has field filter.

  

  Tony McCrae

  committed
  cb45c8a

  Copy full SHA for cb45c8a

• hierarchy field condition.

  

  Tony McCrae

  committed
  faad5b2

  Copy full SHA for faad5b2

• Clean up; println

  

  Tony McCrae

  committed
  8facaf3

  Copy full SHA for 8facaf3

Commits on Jan 9, 2019

• Port any field query builder.

  

  Tony McCrae

  committed
  beb7566

  Copy full SHA for beb7566

• Implement multiple fields query support in elastic6.

  Show description for 8dab8af
  

  Tony McCrae

  committed
  8dab8af

  Copy full SHA for 8dab8af

• Clean up; all make query bit cases implemented so we cab remove the not implemented back stop

  

  Tony McCrae

  committed
  23592e3

  Copy full SHA for 23592e3

• Clean up; on reading the source code a builder action for type field does exist in elastic4s and is called matchType.

  

  Tony McCrae

  committed
  0b22437

  Copy full SHA for 0b22437

• Port nested queries support to elastic6 query builder.

  

  Tony McCrae

  committed
  3ea0016

  Copy full SHA for 3ea0016

• Remove unused InvalidQuery case class which is superseded by the copy in lib query syntax.

  

  Tony McCrae

  committed
  6fb82b9

  Copy full SHA for 6fb82b9

• Clean ups; visually more aligned with the ES1 implementation.

  

  Tony McCrae

  committed
  8e307de

  Copy full SHA for 8e307de

• Refactor; extract match fields definition so that it can be tied across ES1 and ES6 implementations.

  

  Tony McCrae

  committed
  dfa2569

  Copy full SHA for dfa2569

• Marking private to sorts fields and methods.

  Show description for 8aeeed7
  

  Tony McCrae

  committed
  8aeeed7

  Copy full SHA for 8aeeed7

• WIP pick up pagination and default sort order.

  

  Tony McCrae

  committed
  29645d5

  Copy full SHA for 29645d5

• elastic4s can take a list of sorts which removes our responsibility for composing the subsorts ourselves.

  

  Tony McCrae

  committed
  c46fe66

  Copy full SHA for c46fe66

• Added to collection date sort special case. Regression - the Elastic1 sort code mutates the query definition at this point. We'll need to pull this special case up.

  

  Tony McCrae

  committed
  e7e56c9

  Copy full SHA for e7e56c9

• WIP Refactor; kick the date added to collection sort can further down the road by pushing knowledge if it's usage up.

  

  Tony McCrae

  committed
  3b05d99

  Copy full SHA for 3b05d99

• Restore elastic1 dateAddedToCollection sort restrictions by applying as an additional filter.

  

  Tony McCrae

  committed
  158f955

  Copy full SHA for 158f955

• Applies thrall style executeAndLog behaviour to the MediaAPI search calls.

  Show description for ad2e160
  

  Tony McCrae

  committed
  ad2e160

  Copy full SHA for ad2e160

• Image search metrics (time taken as per elastic1).

  

  Tony McCrae

  committed
  107751c

  Copy full SHA for 107751c

• Clean up; elastic6 executions should be bring your own execution context.

  

  Tony McCrae

  committed
  5bf7f2f

  Copy full SHA for 5bf7f2f

• Implement ES6 getImageById

  

  Tony McCrae

  committed
  3e54044

  Copy full SHA for 3e54044

• Initial attempt at porting an aggregation related end point to Elastic6.

  

  Tony McCrae

  committed
  2742139

  Copy full SHA for 2742139

• Implement date histogram based end point.

  Show description for 36fbfd6
  

  Tony McCrae

  committed
  36fbfd6

  Copy full SHA for 36fbfd6

Commits on Jan 10, 2019

• WIP Test for supplier usages end point.

  Show description for aa892ae
  

  Tony McCrae

  committed
  aa892ae

  Copy full SHA for aa892ae

• Developer experience; fail Elastic1 tests if the tear down fails.

  Show description for b4f9ce5
  

  Tony McCrae

  committed
  b4f9ce5

  Copy full SHA for b4f9ce5

• Stable purge test images for ES1 index.

  

  Tony McCrae

  committed
  d4c2f8a

  Copy full SHA for d4c2f8a

• Clean up; remove duplication of ES6 execution.

  

  Tony McCrae

  committed
  a0c9e3e

  Copy full SHA for a0c9e3e

• Additional asserts to document our understanding of deny leases involvement in blocked syndication status.

  

  Tony McCrae

  committed
  6cc39da

  Copy full SHA for 6cc39da

• With some more understanding of what the syndication filter tests are doing we can game our agency test data to stay out of the way.

  Show description for 58a66f5
  

  Tony McCrae

  committed
  58a66f5

  Copy full SHA for 58a66f5

• ES6 implementation for usageForSupplier.

  Show description for bb27656
  

  Tony McCrae

  committed
  bb27656

  Copy full SHA for bb27656

• Naive implementation of suggest completions.

  Show description for 920e892
  

  Tony McCrae

  committed
  920e892

  Copy full SHA for 920e892

• Marking observed bug.

  

  Tony McCrae

  committed
  6e6344b

  Copy full SHA for 6e6344b

• Elastic6 bug fix; missing mappings.

  

  Tony McCrae

  committed
  9bb534c

  Copy full SHA for 9bb534c

• Elastic6 bug fix; missing mappings.

  

  Tony McCrae

  committed
  57f22ce

  Copy full SHA for 57f22ce

Commits on Jan 11, 2019

• Ignore empty suggestions requests.

  Show description for 1738c17
  

  Tony McCrae

  committed
  1738c17

  Copy full SHA for 1738c17

• Restore test image purge.

  

  Tony McCrae

  committed
  41b37bc

  Copy full SHA for 41b37bc

• Revert: usages nested mapping caused a regression in the syndication status tests.

  

  Tony McCrae

  committed
  bc4db28

  Copy full SHA for bc4db28

• Elastic6 nested fields lack the include_in_parent feature which ES1 is probably using to filter for images with instances of certain platform or status (maybe more). Elastic docs suggest that we sh…

  Show description for 641d555
  

  Tony McCrae

  committed
  641d555

  Copy full SHA for 641d555

• Logging; replicas and shards are index (rather than cluster) concerns in Elastic6; hence they're interesting enough to log.

  

  Tony McCrae

  committed
  0315013

  Copy full SHA for 0315013

• Refactor; pull up what's required to config Elasticsearch documentation classes from thrall to common for application to media-api.

  

  Tony McCrae

  committed
  a93e8c1

  Copy full SHA for a93e8c1

• Setting up for switchable media API elastic version; use the ES version interface and see if the compiler finds anything.

  

  Tony McCrae

  committed
  2880e2c

  Copy full SHA for 2880e2c

• Restoring usages as a nested field regressed the can delete check. exists doesn't do what it says in the docs when dealing with a nested field; move to a nested query to try and mitigate.

  

  Tony McCrae

  committed
  01bfb94

  Copy full SHA for 01bfb94

• Media API active elastic version is controller from config.

  

  Tony McCrae

  committed
  2bd58f7

  Copy full SHA for 2bd58f7

• Correct test config.

  

  Tony McCrae

  committed
  c422bef

  Copy full SHA for c422bef

Commits on Jan 14, 2019

• Correct mapping name for usermetaData.subjects.

  Show description for 4a91936
  

  Tony McCrae

  committed
  4a91936

  Copy full SHA for 4a91936

• Test improvement. You can get a speed up by observing that only 1 test actually needs to wait for the image to be indexed; the rest really only need for it to be persisted which is pretty much inst…

  Show description for 835609c
  

  Tony McCrae

  committed
  835609c

  Copy full SHA for 835609c

Commits on Jan 15, 2019

• Test coverage for suspected regression; label edits tend to fail after intial edit.

  Show description for 229f516
  

  Tony McCrae

  committed
  229f516

  Copy full SHA for 229f516

• Log last modified times arriving at update user metadata.

  Show description for 4a3e7cc
  

  Tony McCrae

  committed
  4a3e7cc

  Copy full SHA for 4a3e7cc

• Test coverage for editing of labels.

  

  Tony McCrae

  committed
  6b18510

  Copy full SHA for 6b18510

• Compare usermeta data last modified dates with parsed Java Dates.

  Show description for ce009a9
  

  Tony McCrae

  committed
  ce009a9

  Copy full SHA for ce009a9

• Consistant application of last modified date check; apply to usages as well.

  

  Tony McCrae

  committed
  bbe814c

  Copy full SHA for bbe814c

• Removing TODO; doesn't need to append as the photoshoot suggest is a document level field.

  

  Tony McCrae

  committed
  c4442f8

  Copy full SHA for c4442f8

• Bug fix. Remove duplicate photo shoot suggestions when multiple images have the same photo shoot. Default behaviour changed since Elastic 1.

  Show description for 0aeaffc
  

  Tony McCrae

  committed
  0aeaffc

  Copy full SHA for 0aeaffc

• Aggregation minDocCount=0 should be applied to the date field only as per Elastic1.

  

  Tony McCrae

  committed
  2f16f31

  Copy full SHA for 2f16f31

• Elastic1 used index_name to map edits.labels onto label; suspect that this aggregate is the only usage; could be emulated with a copy_to but if labels is the only usage then we should to remove it …

  Show description for d978de9
  

  Tony McCrae

  committed
  d978de9

  Copy full SHA for d978de9

• Level up. Use Elastic 6.5.4 is the local deploy for consistancy with TEST

  

  Tony McCrae

  committed
  29dc5f2

  Copy full SHA for 29dc5f2

• Clean up. Migrate painless date parsing to Java DataTimeFormatter.ISO_DATE_TIME to remove fragile SimpleDateFormatter.

  

  Tony McCrae

  committed
  1f00f08

  Copy full SHA for 1f00f08

Commits on Jan 16, 2019

• Test fix; incorrecly implemented scroll during purge test images.

  

  Tony McCrae

  committed
  5e15e3d

  Copy full SHA for 5e15e3d

• Spike. Command line tool to migrate images from ES1 to new ES6 index.

  Show description for f5e78e8
  

  Tony McCrae

  committed
  f5e78e8

  Copy full SHA for f5e78e8

• Spike. Copy images from ES1 to ES6. Pass through the Image domain object for validation and JSON cleaning.

  

  Tony McCrae

  committed
  e5fdfb2

  Copy full SHA for e5fdfb2

• WIP. Ignoring XMP metadata data side steps the 1000 field per index limit in default Elastic6.

  Show description for 85699a4
  

  Tony McCrae

  committed
  85699a4

  Copy full SHA for 85699a4

• Bug fix. Incorrecty migrated mappings detected by migration attempt. Print usage pageNumber and syndication expiresOn.

  

  Tony McCrae

  committed
  ab84b88

  Copy full SHA for ab84b88

Commits on Jan 17, 2019

• Override Elastic6 1000 fields limit. The Grid creates a dynamic index field for each and every file metadata field it ever sees.

  Show description for 4113f26
  

  Tony McCrae

  committed
  4113f26

  Copy full SHA for 4113f26

• Override Elastic6 1000 fields limit. The Grid creates a dynamic index field for each and every file metadata field it ever sees.

  Show description for 25008ae
  

  Tony McCrae

  committed
  25008ae

  Copy full SHA for 25008ae

• Disable date detection. Should prevent dynamic fields been defined as date type if the 1st occurrence happens to be a date formatted string. ie. photoshop:DateCreated, exif:DateTimeOriginal

  Show description for e505e93
  

  Tony McCrae

  committed
  e505e93

  Copy full SHA for e505e93

• Increase the max pagination setting from 10k to 100k to accomodate any easy to trigger scrolling error in the grid UI.

  Show description for 6d7c77d
  

  Tony McCrae

  committed
  6d7c77d

  Copy full SHA for 6d7c77d

• Capture the ids of failing migration images.

  

  Tony McCrae

  committed
  64456f7

  Copy full SHA for 64456f7

• Sleep between scrolls to allow single vcpu instances todo something else; like answering health check pings

  

  Tony McCrae

  committed
  9d6b896

  Copy full SHA for 9d6b896

• Bug fix. Incorrecty migrated mappings detected by migration attempt. Print usage pageNumber and syndication expiresOn.

  

  Tony McCrae

  committed
  97335a4

  Copy full SHA for 97335a4

• Disable date detection. Should prevent dynamic fields been defined as date type if the 1st occurrence happens to be a date formatted string. ie. photoshop:DateCreated, exif:DateTimeOriginal

  Show description for 34a39a0
  

  Tony McCrae

  committed
  34a39a0

  Copy full SHA for 34a39a0

• Tweaking bulk import timings.

  

  Tony McCrae

  committed
  b5bf714

  Copy full SHA for b5bf714

Commits on Jan 18, 2019

• Use scan search type as recommended optimisation of Elastic1.7 scroll when order is not important.

  

  Tony McCrae

  committed
  b090f03

  Copy full SHA for b090f03

• Build fix; only apply Main.class farjar setting to migration project.

  

  Tony McCrae

  committed
  f7e2881

  Copy full SHA for f7e2881

• Merge branch 'elastic6-migration' into elastic6-media-api

  

  Tony McCrae

  committed
  ca4d5df

  Copy full SHA for ca4d5df

• Merge branch 'elastic6-media-api' into elastic6

  

  Tony McCrae

  committed
  66e75a4

  Copy full SHA for 66e75a4

• Revert SearchType.SCAN optimisation; setting this resulted in loss of controller of the page size. Seemed to be persisting the pagesize (5000) of a previous scroll.

  

  Tony McCrae

  committed
  cfc5189

  Copy full SHA for cfc5189

• Import Elastic1 source rather than serialized image to migrate the fields the Elastic scripts have been populating (ie. suggesters etc).

  Show description for f00bd64
  

  Tony McCrae

  committed
  f00bd64

  Copy full SHA for f00bd64

• Restore SearchType.Scan optimisation. The page size confusion related to the page size been results returned per shard; hence 5000 was 1000 x 5 shards.

  

  Tony McCrae

  committed
  8301d8c

  Copy full SHA for 8301d8c

• Clean up; be more target with the JSON cleaning we're doing on the suggestMetadataCredit field; use a JSON transformer to document it.

  

  Tony McCrae

  committed
  eec4458

  Copy full SHA for eec4458

• README

  

  Tony McCrae

  committed
  e2b1404

  Copy full SHA for e2b1404

• Lower maximum pagination limit. UI has a limit of 20k so we can come down to just above that.

  

  Tony McCrae

  committed
  aefd997

  Copy full SHA for aefd997

• Clean up; leases.current is marked as unused in the ES1 mappings file; dropping it.

  

  Tony McCrae

  committed
  62b02fd

  Copy full SHA for 62b02fd

• Clean up; leases.current is marked as unused in the ES1 mappings so lets not migrate it.

  

  Tony McCrae

  committed
  300ae51

  Copy full SHA for 300ae51

• Clean up.

  

  Tony McCrae

  committed
  e9a66f0

  Copy full SHA for e9a66f0

• Logging.

  

  Tony McCrae

  committed
  cb7072d

  Copy full SHA for cb7072d

• Merge branch 'master' into elastic6

  

  Tony McCrae

  committed
  e101a80

  Copy full SHA for e101a80

• Tweaking for index speed; replicas = 0 seems like the most impactful change ~ 40%

  

  Tony McCrae

  committed
  2cc439f

  Copy full SHA for 2cc439f

Commits on Jan 21, 2019

• Import tweaks.

  

  Tony McCrae

  committed
  ba03eca

  Copy full SHA for ba03eca

• Merge branch 'master' into elastic6

  

  Tony McCrae

  committed
  9d9262d

  Copy full SHA for 9d9262d

Commits on Jan 22, 2019

• Merge branch 'master' into elastic6

  

  Tony McCrae

  committed
  432ff8a

  Copy full SHA for 432ff8a

Commits on Jan 24, 2019

• Merge branch 'master' into elastic6

  

  Tony McCrae

  committed
  79935ee

  Copy full SHA for 79935ee

Commits on Jan 25, 2019

• Merge branch 'master' into elastic6

  

  Tony McCrae

  committed
  1a88b5c

  Copy full SHA for 1a88b5c