Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

system/build.prop is removed from lineage-15 zips #36

Closed
syphyr opened this issue Oct 29, 2017 · 27 comments
Closed

system/build.prop is removed from lineage-15 zips #36

syphyr opened this issue Oct 29, 2017 · 27 comments

Comments

@syphyr
Copy link
Contributor

syphyr commented Oct 29, 2017

I have build lineage-15.0 and noticed that the generated zip file no longer contains "system/build.prop". This is causing the zip file from showing up on the OTA server. We may no longer count on build.prop being available for parsing.

@julianxhokaxhiu
Copy link
Owner

Is there any replacement for that? Where are the prop stored now?

@syphyr
Copy link
Contributor Author

syphyr commented Oct 29, 2017

The build.prop is only compressed inside the dat file. As far as I can tell, there is no substitute available in uncompressed form.

@syphyr
Copy link
Contributor Author

syphyr commented Nov 5, 2017

Maybe the solution is to use info from the filename if system/build.prop does not exist.

@julianxhokaxhiu
Copy link
Owner

I will try to download one LOS15 ZIP for OP5 and I'll see how the structure is done, and what I can do :) Thanks

@julianxhokaxhiu
Copy link
Owner

I was doing some research and it seems that with the PHP version we will be stuck to the file name ( not very optimal though, we will miss the right way to pull information ).

On the other side, I could rewrite this as a NodeJS app, and there I could use some packages to mount EXT4 system.new.dat and pull the build.prop info.

Unfortunately I see no mod_* or library for PHP that uses FUSE ( there is one, but is very old and unmaintained anymore ).

Any idea?

@julianxhokaxhiu
Copy link
Owner

Inspecting the script, it seems that Google removed this feature: https://github.com/LineageOS/android_build/blob/lineage-15.0/tools/releasetools/ota_from_target_files.py#L394

On cm-14.1 the script was adding the build.prop file.

@syphyr
Copy link
Contributor Author

syphyr commented Nov 6, 2017

It might be pretty memory intensive if system.new.dat is extracted to read the build.prop.

I asked the guys at lineage if they would consider adding that system/build.prop back into lineage-15.0 builds and it does not seem likely. It does not seem to be used by anyone other than us.

@julianxhokaxhiu
Copy link
Owner

I see...this sounds like a blocker to me. Unfortunately not all the infos could be extrapolated from the filename ( see https://github.com/julianxhokaxhiu/LineageOTA/blob/2.0/src/Helpers/Build.php#L76 ) so this means the end of this project basically.

Unless we figure out a way to get the file back somehow, even by reading everytime the system.new.dat ( which would be once per build, as we cache it through memcache ). But maybe would be better proposing the LOS team to re-add the file ( or find a way to patch the script to at least kick this through an environment flag ).

I don't have enough knowledge on Android ROM build, but I figured out that the python script can be patched to maybe obtain that. Any help on this may be useful. Is it worth it to be posted on XDA?

@syphyr
Copy link
Contributor Author

syphyr commented Nov 6, 2017

The only info that we can't get from the filename is ro.build.version.incremental, but do we really need that? Maybe just for delta updates.

ro.build.date.utc could be the actual modified date of the file
we can figure out ro.build.version.sdk from the version of lineage.
we also know ro.lineage.device from the filename.

@julianxhokaxhiu
Copy link
Owner

We can try that but we will end up in the mess of different filenames, per release...not sure if you ever compared ZIP filename since cm9 they changed a lot. And now we have LOS filenames too...

It is possible yes, but not ideal. I'll think about it. Maybe I'll use it as a fallback solution if the build.prop file can't be located.

@syphyr
Copy link
Contributor Author

syphyr commented Nov 7, 2017

During the build process, there is a build.prop in $PROJECT_DIR/out/target/product/$ANDROID_DEVICE/. I could always copy that build.prop into the OTA server directory with the same name as the zip file and the OTA could look for that build.prop file if system/build.prop does not exist. When my script copies the files to the OTA server directory, I could easily include a build.prop that way.

Example:

lineage-15.0-20171030-NIGHTLY-gts210vewifi.zip
lineage-15.0-20171030-NIGHTLY-gts210vewifi.prop

@julianxhokaxhiu
Copy link
Owner

Nice, i like this. Definitely a way to go. We just need to document it and of course apply the logic.

I'll do it later :) thanks!

@julianxhokaxhiu
Copy link
Owner

julianxhokaxhiu commented Nov 10, 2017

@syphyr may I please ask you to try it? You must have at least these two files inside builds/full directory:

$ cd builds/full
$ tree
.
├── lineage-15.0-20171030-NIGHTLY-gts210vewifi.zip
└── lineage-15.0-20171030-NIGHTLY-gts210vewifi.zip.prop

You can checkout the code from here. There is still no release tagged.

Thank you in advance :)

@syphyr
Copy link
Contributor Author

syphyr commented Nov 10, 2017

I have rebased your latest commit and copied the zip and zip.prop files to builds/full. When I run the ota unit test, I only see the cm-14.1 build, but not the lineage-15.0 build. Is there a change needed for the ota unit test to work with lineage-15.0 builds?

@julianxhokaxhiu
Copy link
Owner

May I have the URL?

Anyway, I think this path must be changed, at least the model name.

@syphyr
Copy link
Contributor Author

syphyr commented Nov 10, 2017

Its just a local ip address behind a firewall. I have builds for the same device copied to the OTA Server, one is cm-14.1 and the other is lineage-15.0. I've already edited index.js here: https://github.com/syphyr/CyanogenModOTAUnitTest/blob/master/index.js#L76

The cm-14.1 build shows up in the unit test, but the lineage-15.0 build does not.

Only in /var/www/html/CyanogenModOTA/builds/full: cm-11-20171110-NIGHTLY-p3110.html
Only in /var/www/html/CyanogenModOTA/builds/full: cm-11-20171110-NIGHTLY-p3110.zip
Only in /var/www/html/CyanogenModOTA/builds/full: lineage-14.1-20171108-NIGHTLY-gts210vewifi.html
Only in /var/www/html/CyanogenModOTA/builds/full: lineage-14.1-20171108-NIGHTLY-gts210vewifi.zip
Only in /var/www/html/CyanogenModOTA/builds/full: lineage-14.1-20171108-NIGHTLY-gts210vewifi.zip.md5sum
Only in /var/www/html/CyanogenModOTA/builds/full: lineage-15.0-20171030-NIGHTLY-gts210vewifi.html
Only in /var/www/html/CyanogenModOTA/builds/full: lineage-15.0-20171030-NIGHTLY-gts210vewifi.zip
Only in /var/www/html/CyanogenModOTA/builds/full: lineage-15.0-20171030-NIGHTLY-gts210vewifi.zip.md5sum
Only in /var/www/html/CyanogenModOTA/builds/full: lineage-15.0-20171030-NIGHTLY-gts210vewifi.zip.prop

@julianxhokaxhiu
Copy link
Owner

May I ask you to upload a copy of your build.prop file? Is it existing next to your ZIP file? Is it named the right way as I've shown in the example?

@syphyr
Copy link
Contributor Author

syphyr commented Nov 10, 2017

Here is a copy of my build.prop (I had to zip it up to post it). https://androidfilehost.com/?fid=889964283620765041

Just unzip it and it should be ok. I added the file list of /var/www/html/CyanogenModOTA/builds/full to my last response, so I think those file names should be correct.

Yes, it is like your example.

@syphyr
Copy link
Contributor Author

syphyr commented Nov 10, 2017

tree
.
├── cm-11-20171110-NIGHTLY-p3110.html
├── cm-11-20171110-NIGHTLY-p3110.zip
├── lineage-14.1-20171108-NIGHTLY-gts210vewifi.html
├── lineage-14.1-20171108-NIGHTLY-gts210vewifi.zip
├── lineage-14.1-20171108-NIGHTLY-gts210vewifi.zip.md5sum
├── lineage-15.0-20171030-NIGHTLY-gts210vewifi.html
├── lineage-15.0-20171030-NIGHTLY-gts210vewifi.zip
├── lineage-15.0-20171030-NIGHTLY-gts210vewifi.zip.md5sum
└── lineage-15.0-20171030-NIGHTLY-gts210vewifi.zip.prop

0 directories, 9 files

@julianxhokaxhiu
Copy link
Owner

julianxhokaxhiu commented Nov 10, 2017

Technically it should just work. I see no reason in the code why it should not. I also updated the Unit Test code so it works on top of the official OTA server of LineageOS, although it should work just fine also on top of your own OTA server.

The prop file looks good. It's really difficult to debug it from here :\ Can I please have also your lineage-15.0-20171030-NIGHTLY-gts210vewifi.zip file? So I can simulate it locally. Thanks.

@julianxhokaxhiu
Copy link
Owner

I did one try on my side, and this is what I get:

http://localhost:8000/api/v1/cheeseburger/nightly

{
  "id": null,
  "response": [
    {
      "incremental": "91485dfb93",
      "api_level": "25",
      "url": "http://localhost:8000//builds/full/lineage-14.1-20171106-nightly-cheeseburger-signed.zip",
      "timestamp": "1509963805",
      "md5sum": "8af12aeee3dd912731968b99d156b015",
      "changes": "",
      "channel": "nightly",
      "filename": "lineage-14.1-20171106-nightly-cheeseburger-signed.zip",
      "romtype": "nightly",
      "datetime": "1509963805",
      "version": "14.1",
      "id": "50d3d9b57f33d1ef86abfe88c8e8a0e2c987c8605b717fb148498c4d372d766d"
    },
    {
      "incremental": "",
      "api_level": "",
      "url": "http://localhost:8000//builds/full/lineage-15.0-20171109-nightly-cheeseburger.zip",
      "timestamp": "",
      "md5sum": "f79ce5cd62d1ffada47642ae4a7a58b9",
      "changes": "",
      "channel": "nightly",
      "filename": "lineage-15.0-20171109-nightly-cheeseburger.zip",
      "romtype": "nightly",
      "datetime": "",
      "version": "15.0",
      "id": "d4b45864d9d6fabfc568d74f26c35ababde2105337d7af9a6605e1c56c891aa6"
    }
  ],
  "error": null
}

$ cd builds/full
$ tree
.
├── lineage-14.1-20171106-nightly-cheeseburger-signed.zip
└── lineage-15.0-20171109-nightly-cheeseburger.zip

0 directories, 2 files

I don't have the build.prop file for the LOS15 file as I downloaded one from XDA, although I'm pretty sure you will get a more enriched payload than mine, if it's there.

Please try to download again the patched file ( I fixed another bug that I discovered meanwhile testing ).

@julianxhokaxhiu
Copy link
Owner

Released as 2.7.0. Feel free to use composer to update your code.

@syphyr
Copy link
Contributor Author

syphyr commented Nov 10, 2017

I have updated the OTA Unit Test and I'm still seeing the same results:

nodejs tests/lineage.js

{ id: null,
response:
[ { incremental: '46180fa3e8',
api_level: '25',
url: 'http://192.168.1.5/CyanogenModOTA/builds/full/lineage-14.1-20171108-NIGHTLY-gts210vewifi.zip',
timestamp: '1510175821',
md5sum: '38708015961d63fbb32845e1a5e5d2c1',
changes: 'http://192.168.1.5/CyanogenModOTA/builds/full/lineage-14.1-20171108-NIGHTLY-gts210vewifi.html',
channel: 'nightly',
filename: 'lineage-14.1-20171108-NIGHTLY-gts210vewifi.zip',
romtype: 'nightly',
datetime: '1510175821',
version: '14.1',
id: '86c51c7fd270d738e1f4111dc7ffefe1d2a01b14e9d67494c64cdd437284766f' } ],
error: null }

[email protected] test:cm /usr/local/src/CyanogenModOTAUnitTest
nodejs tests/cm.js

{ id: null,
result:
[ { incremental: '53ed8f53a8',
api_level: '19',
url: 'http://192.168.1.5/CyanogenModOTA/builds/full/cm-11-20171110-NIGHTLY-p3110.zip',
timestamp: '1510332142',
md5sum: '365bc42b251a5f58d3ef0b7b23549713',
changes: 'http://192.168.1.5/CyanogenModOTA/builds/full/cm-11-20171110-NIGHTLY-p3110.html',
channel: 'nightly',
filename: 'cm-11-20171110-NIGHTLY-p3110.zip',
romtype: 'nightly',
datetime: '1510332142',
version: '11',
id: '7fb19a51c93741b7f503a0b6cb7086e74ba2e9f4e8169d1175f5ed79f055b4a6' } ],
error: null }

I have uploaded my lineage-15.0 zip here: https://androidfilehost.com/?fid=962021903579491507

@syphyr
Copy link
Contributor Author

syphyr commented Nov 10, 2017

I rebooted my system and now it shows up. ;-) Its working fine now. Thank you very much for your continued support!

nodejs tests/lineage.js

{ id: null,
response:
[ { incremental: '46180fa3e8',
api_level: '25',
url: 'http://192.168.1.5/CyanogenModOTA/builds/full/lineage-14.1-20171108-NIGHTLY-gts210vewifi.zip',
timestamp: 1510175821,
md5sum: '38708015961d63fbb32845e1a5e5d2c1',
changes: 'http://192.168.1.5/CyanogenModOTA/builds/full/lineage-14.1-20171108-NIGHTLY-gts210vewifi.html',
channel: 'nightly',
filename: 'lineage-14.1-20171108-NIGHTLY-gts210vewifi.zip',
romtype: 'nightly',
datetime: 1510175821,
version: '14.1',
id: '86c51c7fd270d738e1f4111dc7ffefe1d2a01b14e9d67494c64cdd437284766f' },
{ incremental: '',
api_level: '',
url: 'http://192.168.1.5/CyanogenModOTA/builds/full/lineage-15.0-20171030-NIGHTLY-gts210vewifi.zip',
timestamp: 1510083737,
md5sum: '7894de9f9c0386601c2edf0c6e86bf8a',
changes: 'http://192.168.1.5/CyanogenModOTA/builds/full/lineage-15.0-20171030-NIGHTLY-gts210vewifi.html',
channel: 'nightly',
filename: 'lineage-15.0-20171030-NIGHTLY-gts210vewifi.zip',
romtype: 'nightly',
datetime: 1510083737,
version: '15.0',
id: '9c677798867b838a8928ed7118f498e0b698a15c49f5d591aa837dc91f813f22' } ],
error: null }

@syphyr
Copy link
Contributor Author

syphyr commented Nov 10, 2017

One thing I've noticed from the OTA unit test is that "incremental" and "api_level" are unset for lineage-15.0. Does that matter?

@julianxhokaxhiu
Copy link
Owner

I have to check if they are inherited from the build.prop indeed.

Related to the part that it worked after the reboot, it was maybe because of the memcache? Funny although as it worked fine on my side. Whatever, glad that it worked :)

vince2678 pushed a commit to vince2678/LineageOTA that referenced this issue Jun 11, 2018
vince2678 pushed a commit to vince2678/LineageOTA that referenced this issue Jun 11, 2018
@mdPlusPlus
Copy link

Hey, I'm wondering which of the several build.prop files that exist in the lineage folder after building is the right one to push to the LineageOTA server.

$ find -type f -name build.prop
./out/target/product/lavender/obj/PACKAGING/target_files_intermediates/lineage_lavender-target_files-eng.user/VENDOR/odm/etc/build.prop
./out/target/product/lavender/obj/PACKAGING/target_files_intermediates/lineage_lavender-target_files-eng.user/VENDOR/build.prop
./out/target/product/lavender/obj/PACKAGING/target_files_intermediates/lineage_lavender-target_files-eng.user/SYSTEM/product/build.prop
./out/target/product/lavender/obj/PACKAGING/target_files_intermediates/lineage_lavender-target_files-eng.user/SYSTEM/build.prop
./out/target/product/lavender/obj/ETC/system_build_prop_intermediates/build.prop
./out/target/product/lavender/vendor/odm/etc/build.prop
./out/target/product/lavender/vendor/build.prop
./out/target/product/lavender/system/product/build.prop
./out/target/product/lavender/system/build.prop
./cts/tools/release-parser/tests/resources/build.prop`

Can anyone point me in the right direction?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants