Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Release 2.0.4: Fix breaking release 2.0.3 #145

Merged
merged 1 commit into from
Oct 26, 2021
Merged

Release 2.0.4: Fix breaking release 2.0.3 #145

merged 1 commit into from
Oct 26, 2021

Conversation

avahuang0429
Copy link
Contributor

@avahuang0429 avahuang0429 commented Oct 25, 2021

Issue #, if available:
Missed out on one of jackson dependency causing ClassNotFoundException
Description of changes:
Add missed dependency upgrade
Testing:
Ran sample with CLI put-record

dev-dsk-xinlyh-2a-6cde8fe0 % aws kinesis put-record --region us-east-1 --stream-name words --data lobster --partition-key 1
{
    "ShardId": "shardId-000000000000", 
    "SequenceNumber": "49623339037558163172556786039549451903660811450010566658"
}

(21-10-26 16:09:35) <0> [~]  
dev-dsk-xinlyh-2a-6cde8fe0 % aws kinesis put-record --region us-east-1 --stream-name words --data kitty --partition-key 1
{
    "ShardId": "shardId-000000000000", 
    "SequenceNumber": "49623339037558163172556786039550660829480428140769574914"
}

dev-dsk-xinlyh-2a-6cde8fe0 % `amazon_kclpy_helper.py --print_command --java /usr/bin/java --properties samples/sample.properties`
2021-10-26 16:06:48,708 [main] INFO  s.a.k.m.MultiLangDaemonConfig [NONE] - Using a cached thread pool. 
2021-10-26 16:06:48,712 [main] INFO  s.a.k.m.MultiLangDaemonConfig [NONE] - Running PythonKCLSample to process stream words with executable sample_kclpy_app.py 
2021-10-26 16:06:48,713 [main] INFO  s.a.k.m.MultiLangDaemonConfig [NONE] - Using workerId: dd6e185f-0880-4941-8e06-369c853be6cd 
2021-10-26 16:06:48,714 [main] INFO  s.a.k.m.MultiLangDaemonConfig [NONE] - MultiLangDaemon is adding the following fields to the User Agent: amazon-kinesis-client-library-java amazon-kinesis-multi-lang-daemon/1.0.1 python/3.7 sample_kclpy_app.py 
2021-10-26 16:06:49,176 [main] INFO  s.a.k.l.d.DynamoDBLeaseCoordinator [NONE] - With failover time 10000 ms and epsilon 25 ms, LeaseCoordinator will renew leases every 3308 ms, takeleases every 20050 ms, process maximum of 2147483647 leases and steal 1 lease(s) at a time. 
2021-10-26 16:06:49,181 [main] INFO  s.a.k.l.d.DynamoDBLeaseCoordinator [NONE] - With failover time 10000 ms and epsilon 25 ms, LeaseCoordinator will renew leases every 3308 ms, takeleases every 20050 ms, process maximum of 2147483647 leases and steal 1 lease(s) at a time. 
2021-10-26 16:06:49,187 [multi-lang-daemon-0000] INFO  s.a.kinesis.coordinator.Scheduler [NONE] - Initialization attempt 1 
2021-10-26 16:06:49,187 [multi-lang-daemon-0000] INFO  s.a.kinesis.coordinator.Scheduler [NONE] - Initializing LeaseCoordinator 
2021-10-26 16:06:50,276 [multi-lang-daemon-0000] INFO  s.a.kinesis.coordinator.Scheduler [NONE] - Lease table is still empty. Checking again in 3000 ms 
2021-10-26 16:06:53,363 [multi-lang-daemon-0000] INFO  s.a.kinesis.coordinator.Scheduler [NONE] - Lease table is still empty. Checking again in 3000 ms 
2021-10-26 16:06:56,451 [multi-lang-daemon-0000] INFO  s.a.kinesis.coordinator.Scheduler [NONE] - Lease table is still empty. Checking again in 3000 ms 
2021-10-26 16:06:59,540 [multi-lang-daemon-0000] INFO  s.a.kinesis.coordinator.Scheduler [NONE] - Lease table is still empty. Checking again in 3000 ms 
2021-10-26 16:07:02,626 [multi-lang-daemon-0000] INFO  s.a.kinesis.coordinator.Scheduler [NONE] - Lease table is still empty. Checking again in 3000 ms 
2021-10-26 16:07:05,713 [multi-lang-daemon-0000] INFO  s.a.kinesis.coordinator.Scheduler [NONE] - Lease table is still empty. Checking again in 3000 ms 
2021-10-26 16:07:08,713 [multi-lang-daemon-0000] INFO  s.a.kinesis.coordinator.Scheduler [NONE] - Worker dd6e185f-0880-4941-8e06-369c853be6cd is initiating the lease sync. 
2021-10-26 16:07:08,713 [multi-lang-daemon-0000] INFO  s.a.k.c.PeriodicShardSyncManager [NONE] - Syncing Kinesis shard info for words 
2021-10-26 16:07:08,813 [multi-lang-daemon-0000] INFO  s.a.k.leases.KinesisShardDetector [NONE] - Stream words: listing shards with list shards request ListShardsRequest(StreamName=words, ShardFilter=ShardFilter(Type=AT_TRIM_HORIZON)) 
2021-10-26 16:08:09,795 [multi-lang-daemon-0000] INFO  s.a.k.leases.LeaseCleanupManager [NONE] - Starting lease cleanup thread. 
2021-10-26 16:08:09,797 [multi-lang-daemon-0000] INFO  s.a.kinesis.coordinator.Scheduler [NONE] - Starting LeaseCoordinator 
2021-10-26 16:08:09,797 [pool-15-thread-1] INFO  s.a.k.leases.LeaseCleanupManager [NONE] - Number of pending leases to clean before the scan : 0 
2021-10-26 16:08:10,115 [multi-lang-daemon-0000] INFO  s.a.kinesis.coordinator.Scheduler [NONE] - Scheduling periodicShardSync 
2021-10-26 16:08:10,116 [multi-lang-daemon-0000] INFO  s.a.kinesis.coordinator.Scheduler [NONE] - Initialization complete. Starting worker loop. 
2021-10-26 16:08:10,197 [LeaseCoordinator-0000] INFO  s.a.k.l.dynamodb.DynamoDBLeaseTaker [NONE] - Taking leases that have been expired for a long time: [Lease(leaseKey=shardId-000000000000, leaseOwner=null, leaseCounter=0, concurrencyToken=null, lastCounterIncrementNanos=0, checkpoint={SequenceNumber: TRIM_HORIZON,SubsequenceNumber: 0}, pendingCheckpoint=null, pendingCheckpointState=null, ownerSwitchesSinceCheckpoint=0, parentShardIds=[], childShardIds=[], hashKeyRangeForLease=HashKeyRangeForLease(startingHashKey=0, endingHashKey=340282366920938463463374607431768211455))] 
2021-10-26 16:08:10,289 [LeaseCoordinator-0000] INFO  s.a.k.l.dynamodb.DynamoDBLeaseTaker [NONE] - Worker dd6e185f-0880-4941-8e06-369c853be6cd successfully took 1 leases: shardId-000000000000 
2021-10-26 16:08:10,357 [multi-lang-daemon-0000] INFO  s.a.k.c.DeterministicShuffleShardSyncLeaderDecider [NONE] - Elected leaders: dd6e185f-0880-4941-8e06-369c853be6cd 
2021-10-26 16:08:10,358 [multi-lang-daemon-0000] INFO  s.a.k.c.DiagnosticEventLogger [NONE] - Current thread pool executor state: ExecutorStateEvent(executorName=SchedulerThreadPoolExecutor, currentQueueSize=0, activeThreads=0, coreThreads=0, leasesOwned=1, largestPoolSize=0, maximumPoolSize=2147483647) 
2021-10-26 16:08:11,359 [multi-lang-daemon-0000] INFO  s.a.kinesis.coordinator.Scheduler [NONE] - Current stream shard assignments: shardId-000000000000 
2021-10-26 16:08:11,573 [multi-lang-daemon-0000] INFO  s.a.k.r.f.FanOutConsumerRegistration [NONE] - words : StreamConsumer not found, need to create it. 
2021-10-26 16:08:11,770 [multi-lang-daemon-0000] INFO  s.a.k.r.f.FanOutConsumerRegistration [NONE] - words : Waiting for StreamConsumer PythonKCLSample to have ACTIVE status... 
2021-10-26 16:08:12,851 [multi-lang-daemon-0000] INFO  s.a.k.r.f.FanOutConsumerRegistration [NONE] - words : Waiting for StreamConsumer PythonKCLSample to have ACTIVE status... 
2021-10-26 16:08:13,931 [multi-lang-daemon-0000] INFO  s.a.k.r.f.FanOutConsumerRegistration [NONE] - words : Waiting for StreamConsumer PythonKCLSample to have ACTIVE status... 
2021-10-26 16:08:15,010 [multi-lang-daemon-0000] INFO  s.a.k.r.f.FanOutConsumerRegistration [NONE] - words : Waiting for StreamConsumer PythonKCLSample to have ACTIVE status... 
2021-10-26 16:08:16,089 [multi-lang-daemon-0000] INFO  s.a.k.r.f.FanOutConsumerRegistration [NONE] - words : Waiting for StreamConsumer PythonKCLSample to have ACTIVE status... 
2021-10-26 16:08:17,106 [multi-lang-daemon-0000] INFO  s.a.kinesis.coordinator.Scheduler [NONE] - Created new shardConsumer for : ShardInfo(streamIdentifierSerOpt=Optional.empty, shardId=shardId-000000000000, concurrencyToken=29512f3c-519e-49b7-85c2-966d9d448bdf, parentShardIds=[], checkpoint={SequenceNumber: TRIM_HORIZON,SubsequenceNumber: 0}) 
2021-10-26 16:08:17,107 [multi-lang-daemon-0000] INFO  s.a.kinesis.coordinator.Scheduler [NONE] - Sleeping ... 
2021-10-26 16:08:17,109 [ShardRecordProcessor-0000] INFO  s.a.k.l.BlockOnParentShardTask [NONE] - No need to block on parents [] of shard shardId-000000000000 
2021-10-26 16:08:18,199 [multi-lang-daemon-0001] INFO  s.a.kinesis.multilang.LineReaderTask [NONE] - Starting: Reading STDERR for shardId-000000000000 
2021-10-26 16:08:18,200 [ShardRecordProcessor-0000] INFO  s.a.kinesis.multilang.MessageWriter [NONE] - Writing InitializeMessage to child process for shard shardId-000000000000 
2021-10-26 16:08:18,223 [multi-lang-daemon-0002] INFO  s.a.kinesis.multilang.MessageWriter [NONE] - Message size == 110 bytes for shard shardId-000000000000 
2021-10-26 16:08:18,223 [multi-lang-daemon-0002] INFO  s.a.kinesis.multilang.LineReaderTask [NONE] - Starting: Reading next message from STDIN for shardId-000000000000 
2021-10-26 16:08:18,231 [ShardRecordProcessor-0000] INFO  s.a.k.multilang.MultiLangProtocol [NONE] - Received response {"action":"status","responseFor":"initialize"} from subprocess while waiting for initialize while processing shard shardId-000000000000 
2021-10-26 16:08:25,345 [ShardRecordProcessor-0001] INFO  s.a.k.leases.KinesisShardDetector [NONE] - Stream words: listing shards with list shards request ListShardsRequest(StreamName=words) 
2021-10-26 16:08:41,165 [multi-lang-daemon-0000] INFO  s.a.k.c.DiagnosticEventLogger [NONE] - Current thread pool executor state: ExecutorStateEvent(executorName=SchedulerThreadPoolExecutor, currentQueueSize=0, activeThreads=0, coreThreads=0, leasesOwned=1, largestPoolSize=2, maximumPoolSize=2147483647) 
2021-10-26 16:09:09,797 [pool-15-thread-1] INFO  s.a.k.leases.LeaseCleanupManager [NONE] - Number of pending leases to clean before the scan : 0 
2021-10-26 16:09:10,116 [pool-14-thread-1] INFO  s.a.k.c.PeriodicShardSyncManager [NONE] - WorkerId dd6e185f-0880-4941-8e06-369c853be6cd is leader, running the periodic shard sync task 
2021-10-26 16:09:10,198 [pool-14-thread-1] INFO  s.a.k.c.PeriodicShardSyncManager [NONE] - Skipping shard sync for words due to the reason - Hash Ranges are complete for words 
2021-10-26 16:09:10,436 [pool-13-thread-1] INFO  s.a.k.c.DeterministicShuffleShardSyncLeaderDecider [NONE] - Elected leaders: dd6e185f-0880-4941-8e06-369c853be6cd 
2021-10-26 16:09:11,204 [multi-lang-daemon-0000] INFO  s.a.k.c.DiagnosticEventLogger [NONE] - Current thread pool executor state: ExecutorStateEvent(executorName=SchedulerThreadPoolExecutor, currentQueueSize=0, activeThreads=0, coreThreads=0, leasesOwned=1, largestPoolSize=2, maximumPoolSize=2147483647) 
2021-10-26 16:09:18,207 [multi-lang-daemon-0000] INFO  s.a.kinesis.coordinator.Scheduler [NONE] - Current stream shard assignments: shardId-000000000000 
2021-10-26 16:09:18,207 [multi-lang-daemon-0000] INFO  s.a.kinesis.coordinator.Scheduler [NONE] - Sleeping ... 
2021-10-26 16:09:35,088 [ShardRecordProcessor-0001] INFO  s.a.kinesis.multilang.MessageWriter [NONE] - Writing ProcessRecordsMessage to child process for shard shardId-000000000000 
2021-10-26 16:09:35,098 [multi-lang-daemon-0004] INFO  s.a.kinesis.multilang.LineReaderTask [NONE] - Starting: Reading next message from STDIN for shardId-000000000000 
2021-10-26 16:09:35,099 [multi-lang-daemon-0003] INFO  s.a.kinesis.multilang.MessageWriter [NONE] - Message size == 265 bytes for shard shardId-000000000000 
2021-10-26 16:09:35,180 [ShardRecordProcessor-0001] INFO  s.a.kinesis.multilang.MessageWriter [NONE] - Writing CheckpointMessage to child process for shard shardId-000000000000 
2021-10-26 16:09:35,181 [multi-lang-daemon-0004] INFO  s.a.kinesis.multilang.MessageWriter [NONE] - Message size == 134 bytes for shard shardId-000000000000 
2021-10-26 16:09:35,181 [multi-lang-daemon-0004] INFO  s.a.kinesis.multilang.LineReaderTask [NONE] - Starting: Reading next message from STDIN for shardId-000000000000 
2021-10-26 16:09:35,182 [ShardRecordProcessor-0001] INFO  s.a.k.multilang.MultiLangProtocol [NONE] - Received response {"action":"status","responseFor":"processRecords"} from subprocess while waiting for processRecords while processing shard shardId-000000000000 
2021-10-26 16:09:41,283 [multi-lang-daemon-0000] INFO  s.a.k.c.DiagnosticEventLogger [NONE] - Current thread pool executor state: ExecutorStateEvent(executorName=SchedulerThreadPoolExecutor, currentQueueSize=0, activeThreads=0, coreThreads=0, leasesOwned=1, largestPoolSize=2, maximumPoolSize=2147483647) 
2021-10-26 16:10:06,555 [ShardRecordProcessor-0001] INFO  s.a.kinesis.multilang.MessageWriter [NONE] - Writing ProcessRecordsMessage to child process for shard shardId-000000000000 
2021-10-26 16:10:06,555 [multi-lang-daemon-0003] INFO  s.a.kinesis.multilang.LineReaderTask [NONE] - Starting: Reading next message from STDIN for shardId-000000000000 
2021-10-26 16:10:06,555 [multi-lang-daemon-0004] INFO  s.a.kinesis.multilang.MessageWriter [NONE] - Message size == 261 bytes for shard shardId-000000000000 
2021-10-26 16:10:06,556 [ShardRecordProcessor-0001] INFO  s.a.k.multilang.MultiLangProtocol [NONE] - Received response {"action":"status","responseFor":"processRecords"} from subprocess while waiting for processRecords while processing shard shardId-000000000000 
2021-10-26 16:10:09,797 [pool-15-thread-1] INFO  s.a.k.leases.LeaseCleanupManager [NONE] - Number of pending leases to clean before the scan : 0 
2021-10-26 16:10:11,360 [multi-lang-daemon-0000] INFO  s.a.k.c.DiagnosticEventLogger [NONE] - Current thread pool executor state: ExecutorStateEvent(executorName=SchedulerThreadPoolExecutor, currentQueueSize=0, activeThreads=0, coreThreads=0, leasesOwned=1, largestPoolSize=2, maximumPoolSize=2147483647) 
2021-10-26 16:10:19,363 [multi-lang-daemon-0000] INFO  s.a.kinesis.coordinator.Scheduler [NONE] - Current stream shard assignments: shardId-000000000000 
2021-10-26 16:10:19,364 [multi-lang-daemon-0000] INFO  s.a.kinesis.coordinator.Scheduler [NONE] - Sleeping ... 
^C2021-10-26 16:10:31,006 [multi-lang-daemon-0001] ERROR s.a.k.multilang.DrainChildSTDERRTask [NONE] - Received error line from subprocess [Record (Partition Key: 1, Sequence Number: 49623339037558163172556786039549451903660811450010566658, Subsequence Number: 0, Data Size: 7Record (Partition Key: 1, Sequence Number: 49623339037558163172556786039550660829480428140769574914, Subsequence Number: 0, Data Size: 5Traceback (most recent call last):] for shard shardId-000000000000 
Record (Partition Key: 1, Sequence Number: 49623339037558163172556786039549451903660811450010566658, Subsequence Number: 0, Data Size: 7Record (Partition Key: 1, Sequence Number: 49623339037558163172556786039550660829480428140769574914, Subsequence Number: 0, Data Size: 5Traceback (most recent call last):
2021-10-26 16:10:31,006 [multi-lang-daemon-0001] ERROR s.a.k.multilang.DrainChildSTDERRTask [NONE] - Received error line from subprocess [  File "/tmp/kclpy-sample-env/bin/sample_kclpy_app.py", line 161, in <module>] for shard shardId-000000000000 
  File "/tmp/kclpy-sample-env/bin/sample_kclpy_app.py", line 161, in <module>
2021-10-26 16:10:31,006 [multi-lang-daemon-0001] ERROR s.a.k.multilang.DrainChildSTDERRTask [NONE] - Received error line from subprocess [    kcl_process.run()] for shard shardId-000000000000 
    kcl_process.run()
2021-10-26 16:10:31,006 [multi-lang-daemon-0001] ERROR s.a.k.multilang.DrainChildSTDERRTask [NONE] - Received error line from subprocess [  File "/tmp/kclpy-sample-env/lib/python3.7/site-packages/amazon_kclpy/kcl.py", line 289, in run] for shard shardId-000000000000 
  File "/tmp/kclpy-sample-env/lib/python3.7/site-packages/amazon_kclpy/kcl.py", line 289, in run
2021-10-26 16:10:31,006 [Thread-2] INFO  s.a.k.multilang.MultiLangDaemon [NONE] - Process terminated, will initiate shutdown.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@avahuang0429 avahuang0429 changed the title Fix missed dependency upgrade Release 2.0.4: Fix breaking release 2.0.3 Oct 26, 2021
@avahuang0429 avahuang0429 force-pushed the 2.0.3 branch 2 times, most recently from e3ad7cd to 9ed707a Compare October 26, 2021 17:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants