From 560f1ca9a5568f3cb41a9541e7610af10e91483a Mon Sep 17 00:00:00 2001 From: schmonz Date: Sun, 14 Jan 2024 20:07:44 +0000 Subject: [PATCH] tlsh: update to 4.8.2. Changes: 3.5.0: - Added the - force option - Allows a user to force the generation of digests for strings down to 50 characters long 3.5.1: - Fixed the error in the Python extension 3.5.2: - Added the BlackHat Asia tool (presented at Arsenal) 3.7.0: - merged in various fixes - ifdef for SPARC and RH73 - corrected TLSH_CTC_final.pdf - added a SHA1 to the NOTICE.txt file - improved the make.sh so that it calls the test.sh (and does regression tests) - improved regression tests to confirm that the hash is calculated correctly in your environment - fixed the header file C++ standard violation (reserved identifier violation #21) 3.7.1: - resolved issue #29 - the force option for Python Step 1 - adding a regression test for strings approx of length 50 Step 2 - add python code 3.7.2: - added code to set the distance parameters for ROC analysis 3.7.3: - resolving issue #44 - making static library the default 3.7.4: - resolving issue #45 - add a timing test for TLSH 3.7.5: - resolving issue #46 - in include/tlsh_impl.h #define SLIDING_WND_SIZE 5 this can be varied between 4 to 8 3.8.0: - Adding // access functions - required by tools using TLSH library - int Lvalue(); - int Q1ratio(); - int Q2ratio(); 3.9.0: - resolving issue #48 - tlsh_pattern program 3.9.1: - resolving issue #38 - putting in fix in rand_tags.cpp so that it generates identical output to previous version while safely working with pointers 3.9.2: - Also merged the contents of NOTICE.txt into LICENSE. This was done because NOTICE.txt is sometimes accidently removed when people clone this repository. And the LICENSE specifically states that NOTICE.txt should NOT be removed. - Also added command line option -notice which displays the NOTICE.txt file 3.9.3: - currently tlsh_pattern returns all the matches modify tlsh_pattern to return the best match - remove the newline from the input fields when reading in the tlsh_pattern file 3.9.4: - check in order_bug program which demonstrates issue #50 - resolved issue #50 - added code to tlsh_impl.cpp to check for invalid call sequences to update() and final() 3.9.5: - issue #61: added a command line option -notest - do not do any testing 3.9.6: - Have a cmake option to build tlsh with a zero byte checksum (development / research option) - Default build has 1 byte checksum - which is strongly recommended 3.9.7: - resolving issue #50 for bin/timing_unittest 3.9.8: - timing_unittest measures the time taken to do distance calculations - add a command line option -size - so that you can measure the time taken to evaluate different sizes of string 3.9.9: - resolve issue #62 - remove dependancy on GNUInstallDirs 3.10.0: - Adding // access function - required by tools using TLSH library - int BucketValue(int bucket); - int Checksum(int k); 3.11.0: - Make calculation of TLSH digests approx 7 times faster (for large files), done by - inline functions - unrolling loops - fixing the -O2 optimization option 3.11.1: - tidy up: 1. use fast_b_mapping() instead of b_mapping() 2. remove declaration of unsigned r which is never used 3. remove #include which is not required 3.12.0: - remove floating point calculations such as log() function use alookup table instead 3.13.0: - .vcproj files and instructions for builing TLSH on Windows using Visual Studio 3.13.1: - fixing setup.py so that you can install Python Extension on Windows 3.14.0: - adding sliding window size to tlsh_version - changing test.sh to read the sliding window size 3.14.1: - fixing error in test script for -xlen option (print statements about considering length were incorrect) - improved test.sh - tests for existance of expected output files 3.15.0: - Refactor code - so that input of directory or digest is in a struct. The code to process input is in library code (input_desc.cpp, shared_file_functions.cpp). The input routines can be used by myultiple programs. Also, preparing for things like csv input files. 3.15.1: - added command line option -help to show full help information 3.15.2: - tlsh_pattern uses refactored code introduced in 3.15.0 3.16.0: - improved tlsh_pattern functionality - added regression tests for tlsh_pattern 3.16.1: - improved tlsh functionality - add options 3.16.2: - added regression tests for 3.16.1 3.17.0: - Make command line option -force (50 char limit) the default behaviour - Add a command line option -conservative (256 char limit) 3.17.3: - add checking to confirm that TLSH digests are the correct length in - -c option - -d option - the appropriate column of -l listfile options 3.18.0: - resolve issue #72 - remove tlsh_version 3.19.0: - preperation for Windows build remove ../Testing/ from test.sh script and from regression test results 3.19.1: - in test.sh and testlen.sh - make TLSH_PROG a variable 4.0.0: - version 4: adding version identifier to each digest: 'T1' - adding command line option -old to generate old style digests - In this version - the showvers is defaulted to off - so this will pass the old regression tests 4.0.1: - turing on T1 functionality by setting showvers=1 in main - updating regression tests to have T1 at the start of digests 4.1.0: - adding -o option for output filename (output will go to stdout if no output file given) - changed test scripts to use -o option - adding -ojson option for json output - added regression test for -ojson option - adding -onull option to output empty files / files too small as TNULL 4.2.0: - Windows version using minGW 4.2.1: - resolve issue #78 json objects do not validate on windows 4.2.2: - resolve issue #81 - Pass regression tests 4.2.3: - add regression tests that are compatible with https://github.com/glaslos/tlsh 4.3.0: - issue #79 - divide by 0 if q3 == 0 solution. if (q3 == 0) return invalid hash 4.4.0: - Fixing Python Extension - updated python extension to T1 hashes (4.0.0) - fixed python_test.sh (which attempted to access old expected results files) - added license information to py_ext/tlshmodule.cpp 4.4.1: - Command line options to tlsh_digest.py -conservative enforce 256 byte limit -old generate old style hash (without "T1") - added python functions to tlsh package (for backwards compatibility) tlsh.oldhash(data) tlsh.conservativehash(data) tlsh.oldconservativehash(data) 4.5.0: - Checking in files to create pypi package 4.6.0: - Add architecture ppc64le to travis build (Thanks ddeka2910) 4.7.0: - Release updated package py-tlsh on Pypi.org - Merging in pull request that adds functions to Python package lvalue, q1ratio, q2ratio, checksum, bucket_value and is_valid - resolve issue #102 - correct Python version numbers 4.7.2: - regression tests for C++ and Python functions for: lvalue, q1ratio, q2ratio, checksum, bucket_value - resolve issue #95 - allow Requires-Python: >=2.7 4.8.0: - Fix the make install target by adding the version.h in the installed files 4.8.1: - Improve portability, add shared library build, install tlsh_unittest 4.8.2: - fixed tlsh_win_version.h --- devel/tlsh/Makefile | 32 +++++++++++-------- devel/tlsh/Makefile.common | 16 ---------- devel/tlsh/PLIST | 12 +++++-- devel/tlsh/buildlink3.mk | 5 ++- devel/tlsh/distinfo | 10 +++--- devel/tlsh/patches/patch-CMakeLists.txt | 17 ++++++++++ devel/tlsh/patches/patch-utils_rand__tags.cpp | 15 --------- 7 files changed, 52 insertions(+), 55 deletions(-) delete mode 100644 devel/tlsh/Makefile.common create mode 100644 devel/tlsh/patches/patch-CMakeLists.txt delete mode 100644 devel/tlsh/patches/patch-utils_rand__tags.cpp diff --git a/devel/tlsh/Makefile b/devel/tlsh/Makefile index 62e22319d1e1..a390ba5e2857 100644 --- a/devel/tlsh/Makefile +++ b/devel/tlsh/Makefile @@ -1,22 +1,28 @@ -# $NetBSD: Makefile,v 1.2 2017/09/01 07:52:02 wiz Exp $ +# $NetBSD: Makefile,v 1.3 2024/01/14 20:07:44 schmonz Exp $ -PKGREVISION= 1 -USE_CMAKE= yes -CONFIGURE_DIRS= build/release -CMAKE_ARG_PATH= ../.. -USE_LANGUAGES= c c++ -# all tests fail -#TEST_TARGET= test +DISTNAME= tlsh-4.8.2 +CATEGORIES= devel +MASTER_SITES= ${MASTER_SITE_GITHUB:=trendmicro/} +GITHUB_TAG= ${PKGVERSION_NOREV} + +MAINTAINER= pkgsrc-users@NetBSD.org +HOMEPAGE= https://github.com/trendmicro/tlsh/ +COMMENT= Fuzzy matching library +LICENSE= apache-2.0 + +USE_CMAKE= yes +CONFIGURE_DIRS= build/release +CMAKE_ARG_PATH= ../.. +CMAKE_ARGS+= -DTLSH_SHARED_LIBRARY=1 +USE_LANGUAGES= c c++ +TEST_TARGET= test INSTALLATION_DIRS= bin include lib post-extract: - mkdir ${WRKSRC}/build/release + ${MKDIR} ${WRKSRC}/${CONFIGURE_DIRS} -do-install: - ${INSTALL_DATA} ${WRKSRC}/bin/tlsh_unittest ${DESTDIR}${PREFIX}/bin +post-install: ${INSTALL_DATA} ${WRKSRC}/include/*.h ${DESTDIR}${PREFIX}/include - ${INSTALL_DATA} ${WRKSRC}/lib/libtlsh* ${DESTDIR}${PREFIX}/lib -.include "Makefile.common" .include "../../mk/bsd.pkg.mk" diff --git a/devel/tlsh/Makefile.common b/devel/tlsh/Makefile.common deleted file mode 100644 index 656841433f4e..000000000000 --- a/devel/tlsh/Makefile.common +++ /dev/null @@ -1,16 +0,0 @@ -# $NetBSD: Makefile.common,v 1.4 2016/09/04 10:41:48 wiz Exp $ -# used by devel/tlsh/Makefile -# used by devel/py-tlsh/Makefile - -DISTNAME= tlsh-3.4.5 -CATEGORIES= devel -MASTER_SITES= ${MASTER_SITE_GITHUB:=trendmicro/} -GITHUB_PROJECT= tlsh -GITHUB_TAG= v${PKGVERSION_NOREV} - -MAINTAINER= pkgsrc-users@NetBSD.org -HOMEPAGE= https://github.com/trendmicro/tlsh/ -COMMENT= Fuzzy matching library -LICENSE= apache-2.0 - -DISTINFO_FILE= ${.CURDIR}/../../devel/tlsh/distinfo diff --git a/devel/tlsh/PLIST b/devel/tlsh/PLIST index 117f5f5e3e5a..643884dde714 100644 --- a/devel/tlsh/PLIST +++ b/devel/tlsh/PLIST @@ -1,10 +1,16 @@ -@comment $NetBSD: PLIST,v 1.2 2016/09/01 17:04:08 wiz Exp $ +@comment $NetBSD: PLIST,v 1.3 2024/01/14 20:07:45 schmonz Exp $ +bin/tlsh bin/tlsh_unittest +include/input_desc.h +include/shared_file_functions.h include/tlsh.h +include/tlsh/tlsh.h +include/tlsh/tlsh_version.h include/tlsh_impl.h include/tlsh_util.h -include/version.h +include/tlsh_version.h +include/tlsh_win_version.h lib/libtlsh.a lib/libtlsh.so -lib/libtlsh.so.0 +lib/libtlsh.so.4 lib/libtlsh.so.${PKGVERSION} diff --git a/devel/tlsh/buildlink3.mk b/devel/tlsh/buildlink3.mk index e9b2be0a9ebb..bb2fcdf67dd5 100644 --- a/devel/tlsh/buildlink3.mk +++ b/devel/tlsh/buildlink3.mk @@ -1,12 +1,11 @@ -# $NetBSD: buildlink3.mk,v 1.1 2015/12/02 17:22:18 wiz Exp $ +# $NetBSD: buildlink3.mk,v 1.2 2024/01/14 20:07:45 schmonz Exp $ BUILDLINK_TREE+= tlsh .if !defined(TLSH_BUILDLINK3_MK) TLSH_BUILDLINK3_MK:= -BUILDLINK_DEPMETHOD.tlsh?= build -BUILDLINK_API_DEPENDS.tlsh+= tlsh>=3.4.3 +BUILDLINK_API_DEPENDS.tlsh+= tlsh>=4.8.2 BUILDLINK_PKGSRCDIR.tlsh?= ../../devel/tlsh .endif # TLSH_BUILDLINK3_MK diff --git a/devel/tlsh/distinfo b/devel/tlsh/distinfo index d81b17c0a90b..2c16ee837742 100644 --- a/devel/tlsh/distinfo +++ b/devel/tlsh/distinfo @@ -1,6 +1,6 @@ -$NetBSD: distinfo,v 1.7 2021/10/26 10:20:04 nia Exp $ +$NetBSD: distinfo,v 1.8 2024/01/14 20:07:45 schmonz Exp $ -BLAKE2s (tlsh-3.4.5.tar.gz) = 8a70191cd5be8f618d881a1927007492bd4b1384320ad39492b71ae3faeff146 -SHA512 (tlsh-3.4.5.tar.gz) = 039f9fb9fb467a9b22f85a621c46069bac1745da039657840e1ff1a53901392fcf29d3c953b0f47d831d022a5e2594925c726d4ff921614418a3bb023a4ebbdc -Size (tlsh-3.4.5.tar.gz) = 1914815 bytes -SHA1 (patch-utils_rand__tags.cpp) = 024d61efd590d5e16b20100eac729276fc996ee6 +BLAKE2s (tlsh-4.8.2.tar.gz) = 846377f453b4365ebcaabe09f22f5579087d50999e7edfc9dc8117277b7cb79e +SHA512 (tlsh-4.8.2.tar.gz) = 5355c7734f5356a3a7a31ccc078254835871fc0369f1889a34131a0f810735201c68f674eb18e975b6a03a32e5c284899ca9ffdc1f583e42192188c172d2aad0 +Size (tlsh-4.8.2.tar.gz) = 3345458 bytes +SHA1 (patch-CMakeLists.txt) = ce5dae73e7955cbf315fa97080390d6aba64f017 diff --git a/devel/tlsh/patches/patch-CMakeLists.txt b/devel/tlsh/patches/patch-CMakeLists.txt new file mode 100644 index 000000000000..13721c60e403 --- /dev/null +++ b/devel/tlsh/patches/patch-CMakeLists.txt @@ -0,0 +1,17 @@ +$NetBSD: patch-CMakeLists.txt,v 1.1 2024/01/14 20:07:45 schmonz Exp $ + +Squelch configure warnings. + +--- CMakeLists.txt.orig 2021-09-09 05:56:28.000000000 +0000 ++++ CMakeLists.txt +@@ -52,6 +52,10 @@ + # OF THE POSSIBILITY OF SUCH DAMAGE. + + cmake_minimum_required(VERSION 2.6) ++# let CMAKE_MODULE_PATH override modules installed with cmake ++cmake_policy(SET CMP0017 OLD) ++# set MACOSX_RPATH ++cmake_policy(SET CMP0042 NEW) + + project(TLSH) + diff --git a/devel/tlsh/patches/patch-utils_rand__tags.cpp b/devel/tlsh/patches/patch-utils_rand__tags.cpp deleted file mode 100644 index d924edbb1048..000000000000 --- a/devel/tlsh/patches/patch-utils_rand__tags.cpp +++ /dev/null @@ -1,15 +0,0 @@ -$NetBSD: patch-utils_rand__tags.cpp,v 1.2 2017/09/01 07:52:02 wiz Exp $ - -https://github.com/trendmicro/tlsh/commit/fb076b0f62b620e0de4cf95a5ccb81f21d4abc24#diff-02ad04ab1753a2119772311c29f24d23 - ---- utils/rand_tags.cpp.orig 2016-12-16 17:30:02.495417195 +0000 -+++ utils/rand_tags.cpp -@@ -366,7 +366,7 @@ struct tagdef oneoff_def[] = { - - static void rhtml_contents(std::string &htmls, int *ntags, int *ndistinct_tags) - { -- if ((*ntags <= 0) && (ndistinct_tags <= 0)) -+ if ((*ntags <= 0) && (*ndistinct_tags <= 0)) - return; - if (random() % 10 == 1) { - anchor(htmls);