[LineageOS][OTA][PHP] Open Source REST Server for you - Android

Hi guys,
today I would like to present to you a simple project that was born on this thread to accomplish a very simple task: since there are thousand of Custom ROMs around here, so much of them are LineageOS based, and because of this their forced to build every time all the ROM and post updates here. Users are also forced to check if their preferred ROM was updated or not. That's why I decided to understand how LineageOS OTA Updater System App works. Because of this I've wrote a simple REST Server API emulation that fully works with your ROM (if integrated of course). How? Continue reading down here.
How does it work?
Of course is as simple as it should be. Clone the repo (is a simple PHP website that you can host on any shared hosting*/VPS that you like) and upload it into your preferred hosting. That's it. If you point your web browser to that address the App is already working.
After, you have to upload all your build to the _builds/ folder, and you're done.
The two already working calls /api and /api/v1/build/get_delta should answer correctly the updater app to make it work (remember that visiting it as a browser is not sufficent).
How to integrate it with my ROM?
You have two options:
- Declare cm.updater.uri on your own build.prop file with the value of you own server URL where you have deployed it (This can be done also by the user with any Android App from the Market)
- Replace the string conf_update_server_url_def value inside values.xml of the OTA App source code (COMPILE TIME ONLY!)
Which builds does it support?
Anything that will be borned by the official guide on how to build your custom LineageOS ROM! So, in poor words will be: stable, rcs, nightly and snapshots (likely called EXPERIMENTAL), from CM7 to CM14.
Delta updates SHOULD work too. Just try it and tell me if they works
How can I debug it?
You can use this simple UnitTest that I've already pushed into my GitHub. Feel free to use it everytime you need. It's based upon NodeJS and Unirest.
Is it free?
"Free software is a matter of liberty, not price. To understand the concept, you should think of free as in free speech, not as in free beer."
—Richard Stallman
Click to expand...
Click to collapse
Use it as you want, do anything you want with it as it's MIT licensed.
Is it free of bugs?
That's why I'm here Only you can help me to squash all the remaining bugs!
I hope this will be useful to anyone of you, helping the ROM community providing a simple OTA updater that already works on LineageOS official ROMs.
Greets.
---
Project Home: https://github.com/julianxhokaxhiu/LineageOTA
More about the study: http://blog.julianxhokaxhiu.com/how-the-cm-ota-server-works-and-how-to-implement-and-use-ours
Changelog Build Scripts ( thanks to @Deltadroid ): https://github.com/syphyr/cm_build_scripts/blob/master/make_changelog
* On a Shared Hosting you can ONLY provide a FULL ROM download, NOT DELTAs!
---
Donators:
- @BlueFlame4 x2

---

Hey, I'm having trouble using your docker image behind a Nginx https reverse proxy. The server is correctly answering the requests:
Code:
{
"id":null,
"response":[
{
"incremental":"",
"api_level":"",
"url":"http:\/\/MYDOMAIN\/\/builds\/full\/lineage-17.1-20200830-UNOFFICIAL-lavender.zip",
"timestamp":1598774045,
"md5sum":"718fb89f935b979edd57b2642234d1fa",
"changes":"",
"channel":"unofficial",
"filename":"lineage-17.1-20200830-UNOFFICIAL-lavender.zip",
"romtype":"unofficial",
"datetime":1598774045,
"version":"17.1",
"id":"50533a894b2ab0d9b2711444ca4f2b530a8ff2389723ea2bd7ada6e029599e2c",
"size":914450521
}
],
"error":null
}
But the returned `url` is http-only and the updater can't download it (throws error). WhenI try to curl it without `-L` I only get `301 Moved Permanently`. Only when I append `-L` I get the correct binary response over https. I'm pretty sure this is a redirecting issue.
Here's my Nginx config:
Code:
server{
server_name MYDOMAIN;
server_tokens off;
listen 80;
listen [::]:80 ipv6only=on;
# Redirect all HTTP requests to HTTPS with a 301 Moved Permanently response.
return 301 https://$host$request_uri;
}
server {
server_name MYDOMAIN;
server_tokens off;
listen 443 ssl http2;
listen [::]:443 ssl http2 ipv6only=on;
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-NginX-Proxy true;
proxy_pass http://127.0.0.1:24087;
proxy_redirect off;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
add_header Strict-Transport-Security "max-age=15768000; includeSubdomains; preload";
add_header Referrer-Policy same-origin;
add_header X-Content-Type-Options nosniff;
add_header X-Frame-Options DENY;
add_header X-Xss-Protection "1; mode=block";
[...]
Any idea what I'm doing wrong?
Basically I want to know how to force the `url` response in the JSON to be `https` instead of `http`. (At least that's what I think is the reason for the updater not being able to download the image.)
Code:
08-30 16:00:19.407 7025 7025 D UpdaterController: Starting 50533a894b2ab0d9b2711444ca4f2b530a8ff2389723ea2bd7ada6e029599e2c
08-30 16:00:19.409 7025 7921 E HttpURLConnectionClient: Error downloading file
08-30 16:00:19.409 7025 7921 E HttpURLConnectionClient: java.io.IOException: Cleartext HTTP traffic to MYDOMAIN not permitted
08-30 16:00:19.409 7025 7921 E HttpURLConnectionClient: at com.android.okhttp.HttpHandler$CleartextURLFilter.checkURLPermitted(HttpHandler.java:124)
08-30 16:00:19.409 7025 7921 E HttpURLConnectionClient: at com.android.okhttp.internal.huc.HttpURLConnectionImpl.execute(HttpURLConnectionImpl.java:462)
08-30 16:00:19.409 7025 7921 E HttpURLConnectionClient: at com.android.okhttp.internal.huc.HttpURLConnectionImpl.connect(HttpURLConnectionImpl.java:131)
08-30 16:00:19.409 7025 7921 E HttpURLConnectionClient: at org.lineageos.updater.download.HttpURLConnectionClient$DownloadThread.run(HttpURLConnectionClient.java:250)
08-30 16:00:19.409 7025 7921 E UpdaterController: Download failed
EDIT: Solved!
Code:
location / {
proxy_pass http://127.0.0.1:24087;
proxy_http_version 1.1;
proxy_cache_bypass $http_upgrade;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
#proxy_set_header Host $host;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-NginX-Proxy true;
}

---

Not sure if this project is still supported, but with build.prop change, it just checks for updates forever on phone. When I try debugging with the UnitTest script, I get "Not Found The requested URL /CyanogenModOTA/api was not found on this server." (definitely server-related) I've gotten this working before on another VPS, so I'm wondering why it's not on my build server.
Are there any specific packages I need to install to get this working?
Here is my php config: http://hongbuild.ddns.net:81/test.php

---

klvnhng said:
Not sure if this project is still supported, but with build.prop change, it just checks for updates forever on phone. When I try debugging with the UnitTest script, I get "Not Found The requested URL /CyanogenModOTA/api was not found on this server." (definitely server-related) I've gotten this working before on another VPS, so I'm wondering why it's not on my build server.
Are there any specific packages I need to install to get this working?
Here is my php config: http://hongbuild.ddns.net:81/test.php
Click to expand...
Click to collapse
Since build.prop is in the system.new.dat file in lollipop builds, I just started copying it over from my build server and modified the Build.php file to look for the file instead of inside the zip. I just got this working on my web server. Only issue I'm having is related to change logs but I'm going to try and figure that one out later.
Here are the changes I did to get this working with CyanogenMod 12 (all changes will be assumed from the CyanogenModOTA directory):
- mkdir -p builds/buildprop <-- this is where you copy your build.prop from each build (they go in a folder with the same name as the build zip e.x. cm-12-20150103-NIGHTLY)
- mkdir -p builds/changelog <-- this will have your change logs (name them the same as the build zip except with a .txt extension)
- update setConfig( 'basePath', 'CyanogenModOTA' ) to setConfig( 'basePath', 'http://wfhome.net/CyanogenModOTA' ) in index.php (that is my server)
- change the Build constructor function to this (I updated the preg_match_all line, added the buildPropFolder variable, updated the $this->buildProp line, and changed the changeLogUrl line):
Code:
private $buildPropFolder = '';
public function __construct($fileName, $physicalPath) {
/*
$tokens Schema:
array(
1 => [CM VERSION] (ex. 10.1.x, 10.2, 11, etc.)
2 => [DATE OF BUILD] (ex. 20140130)
3 => [CHANNEL OF THE BUILD] (ex. RC, RC2, NIGHTLY, etc.)
4 => [MODEL] (ex. i9100, i9300, etc.)
)
*/
preg_match_all( '/cm-([0-9\.]+-)(\d+-)?([a-zA-Z0-9]+-)?([a-zA-Z0-9]+).zip/', $fileName, $tokens );
$tokens = $this->removeTrailingDashes( $tokens );
$this->filePath = $physicalPath . '/' . $fileName;
$this->buildPropFolder = str_replace('/full', '/buildprop', $physicalPath) . '/' . preg_replace('/\\.[^.\\s]{3,4}$/', '', $fileName);
$this->buildProp = explode( "\n", file_get_contents($this->buildPropFolder . '/build.prop') );
$this->channel = $this->_getChannel( str_replace( range( 0 , 9 ), '', $tokens[3] ) );
$this->filename = $fileName;
$this->url = $this->_getUrl( '', Flight::cfg()->get('buildsPath') );
$this->changelogUrl = str_replace('/full', '/changelog', $this->_getChangelogUrl());
$this->timestamp = filemtime( $this->filePath );
$this->incremental = $this->getBuildPropValue( 'ro.build.version.incremental' );
$this->apiLevel = $this->getBuildPropValue( 'ro.build.version.sdk' );
$this->model = $this->getBuildPropValue( 'ro.cm.device' );
}
Also he has memcached setup in there, so you might make sure you have memcache installed and set to run on startup on your server. You also need xdelta3 to create delta builds although, I don't think it is working ( I compiled the latest version of xdelta3 and it doesn't appear to do anything as there are not any files being created in the delta folder).
Hope that helps.

---

Thanks for the help (I'll definitely need it when I actually want to start USING the server), but it seems you've misunderstood me. Right now, I can't even get the rest server running properly-that's why I'm getting a 404 error.
Notice you get an output when you go to http://wfhome.net/CyanogenModOTA/api, I don't get anything!

---

klvnhng said:
Thanks for the help (I'll definitely need it when I actually want to start USING the server), but it seems you've misunderstood me. Right now, I can't even get the rest server running properly-that's why I'm getting a 404 error.
Notice you get an output when you go to http://wfhome.net/CyanogenModOTA/api, I don't get anything!
Click to expand...
Click to collapse
Ah sorry. First thing I see is that you need mod_rewrite installed in apache.
Code:
sudo a2enmod rewrite
sudo service apache2 restart

---

Need help with development
First I would like to thank you for using this product and playing with it (which involves patching, testing, etc.).
Since the first post I've made here in XDA the project evolved a little since I rewrote entirely the project to make it composer friendly. Rather than that, the Delta build process is actually non working (I tried to create a ZIP but I don't have enough knowledge in ROM development to tell if it's enough or not) so it's just a WIP layer that should be addressed and fixed (I've already found a Python project which does this already but I'm of the idea that this should be somehow not be a bloated software that needs to install binaries here and there to make it working).
So, actually I'm not really working in this project but It's in my TODO list, meanwhile I'll be very happy to have pull requests with useful patches to make this KK, LL and more compatible So feel free to contribute and thanks again for using it!

---

JulianXhokaxhiu said:
First I would like to thank you for using this product and playing with it (which involves patching, testing, etc.).
Since the first post I've made here in XDA the project evolved a little since I rewrote entirely the project to make it composer friendly. Rather than that, the Delta build process is actually non working (I tried to create a ZIP but I don't have enough knowledge in ROM development to tell if it's enough or not) so it's just a WIP layer that should be addressed and fixed (I've already found a Python project which does this already but I'm of the idea that this should be somehow not be a bloated software that needs to install binaries here and there to make it working).
So, actually I'm not really working in this project but It's in my TODO list, meanwhile I'll be very happy to have pull requests with useful patches to make this KK, LL and more compatible So feel free to contribute and thanks again for using it!
Click to expand...
Click to collapse
Thank YOU for making it! Really appreciate the time and effort you've put into this project.
rjwil1086 said:
Ah sorry. First thing I see is that you need mod_rewrite installed in apache.
Code:
sudo a2enmod rewrite
sudo service apache2 restart
Click to expand...
Click to collapse
That did it, thanks :good:
I've also implemented your changes, but when I check for updates on my phone, I still get "No new updates found". I've copied my new build over to builds/full, and the build.prop to builds/buildprop/cm*

---

rjwil1086 thank you very much for your help and suggestions, I made my own server http://paksman.ddns.net/cyanogenmodota, edited build.prop for cm updater to look for server url, made all the changes as you but I always get "No new updates found". Not sure if there is a problem with my server or with this project in general. Have you made any success to make this work?

---

I have it working. I'll upload mine to github tonight

---

Think I solved my own problem. My builds were all tagged as 'UNOFFICIAL'. When I changed this to 'NIGHTLY' (for the build and build.prop folder respectively ) they finally started to be recognised by cm updater app. JulianXhokaxhiu and rjwil1086, thank you so much for your effort,your work is being much appreciated.

---

Packsman said:
Think I solved my own problem. My builds were all tagged as 'UNOFFICIAL'. When I changed this to 'NIGHTLY' (for the build and build.prop folder respectively ) they finally started to be recognised by cm updater app. JulianXhokaxhiu and rjwil1086, thank you so much for your effort,your work is being much appreciated.
Click to expand...
Click to collapse
Yup. Sorry. I knew that but forgot to mention it. That's an issue with the CMUpdater app more than it is with the REST implementation. It doesn't look for unofficial files

---

Packsman said:
Think I solved my own problem. My builds were all tagged as 'UNOFFICIAL'. When I changed this to 'NIGHTLY' (for the build and build.prop folder respectively ) they finally started to be recognised by cm updater app. JulianXhokaxhiu and rjwil1086, thank you so much for your effort,your work is being much appreciated.
Click to expand...
Click to collapse
Awesome! This fixed it for me as well. Thanks to everyone for the help

---

[CyanogenMod][OTA][PHP] Open Source REST Server for you
Hi, I have been tested server code with a free host (000webhost) then accessed the web, I received messages:
Warning: Unexpected character in input: '\' (ASCII=92) state=1 in /home/a5885282/public_html/index.php on line 27
Parse error: syntax error, unexpected T_STRING, expecting T_CONSTANT_ENCAPSED_STRING or '(' in /home/a5885282/public_html/index.php on line 27
The line 27 of index.php is "use \JX\CmOta\CmOta;". But I don't know about PHP. Please help me solve this issue.

---

@rjwil1086 , @klvnhng , @Packsman please help me to solve following error:
$ node index.js
<h1>500 Internal Server Error</h1><h3>Undefined offset: 0 (8)</h3><pre>#0 /var/www/CyanogenModOTA/src/Helpers/Build.php(214): flight\Engine->handleError(8, 'Undefined offse...', '/var/www/Cyanog...', 214, Array)
#1 /var/www/CyanogenModOTA/src/Helpers/Build.php(63): JX\CmOta\Helpers\Build->removeTrailingDashes(Array)
#2 /var/www/CyanogenModOTA/src/Helpers/Builds.php(115): JX\CmOta\Helpers\Build->__construct('cm-11-20140103-...', '/var/www/Cyanog...')
#3 /var/www/CyanogenModOTA/src/Helpers/Builds.php(49): JX\CmOta\Helpers\Builds->getBuilds()
#4 /var/www/CyanogenModOTA/vendor/mikecao/flight/flight/core/Loader.php(123): JX\CmOta\Helpers\Builds->__construct()
#5 /var/www/CyanogenModOTA/vendor/mikecao/flight/flight/core/Loader.php(80): flight\core\Loader->newInstance('\JX\CmOta\Helpe...', Array)
#6 /var/www/CyanogenModOTA/vendor/mikecao/flight/flight/Engine.php(69): flight\core\Loader->load('builds', true)
#7 /var/www/CyanogenModOTA/vendor/mikecao/flight/flight/core/Dispatcher.php(191): flight\Engine->__call('builds', Array)
#8 /var/www/CyanogenModOTA/vendor/mikecao/flight/flight/core/Dispatcher.php(191): flight\Engine->builds()
#9 /var/www/CyanogenModOTA/vendor/mikecao/flight/flight/Flight.php(43): flight\core\Dispatcher::invokeMethod(Array, Array)
#10 /var/www/CyanogenModOTA/src/CmOta.php(97): Flight::__callStatic('builds', Array)
#11 /var/www/CyanogenModOTA/src/CmOta.php(97): Flight::builds()
#12 /var/www/CyanogenModOTA/vendor/mikecao/flight/flight/core/Dispatcher.php(160): JX\CmOta\{closure}()
#13 /var/www/CyanogenModOTA/vendor/mikecao/flight/flight/core/Dispatcher.php(143): flight\core\Dispatcher::callFunction(Object(Closure), Array)
#14 /var/www/CyanogenModOTA/vendor/mikecao/flight/flight/Engine.php(310): flight\core\Dispatcher::execute(Object(Closure), Array)
#15 /var/www/CyanogenModOTA/vendor/mikecao/flight/flight/core/Dispatcher.php(191): flight\Engine->_start()
#16 /var/www/CyanogenModOTA/vendor/mikecao/flight/flight/core/Dispatcher.php(142): flight\core\Dispatcher::invokeMethod(Array, Array)
#17 /var/www/CyanogenModOTA/vendor/mikecao/flight/flight/core/Dispatcher.php(48): flight\core\Dispatcher::execute(Array, Array)
#18 /var/www/CyanogenModOTA/vendor/mikecao/flight/flight/Engine.php(64): flight\core\Dispatcher->run('start', Array)
#19 /var/www/CyanogenModOTA/vendor/mikecao/flight/flight/core/Dispatcher.php(191): flight\Engine->__call('start', Array)
#20 /var/www/CyanogenModOTA/vendor/mikecao/flight/flight/core/Dispatcher.php(191): flight\Engine->start()
#21 /var/www/CyanogenModOTA/vendor/mikecao/flight/flight/Flight.php(43): flight\core\Dispatcher::invokeMethod(Array, Array)
#22 /var/www/CyanogenModOTA/src/CmOta.php(80): Flight::__callStatic('start', Array)
#23 /var/www/CyanogenModOTA/src/CmOta.php(80): Flight::start()
#24 /var/www/CyanogenModOTA/index.php(35): JX\CmOta\CmOta->run()
#25 {main}</pre>
Thank you so much.

---

Thanks all, I fixed.

---

[CyanogenMod][OTA][PHP] Open Source REST Server for you
I have just creat susscess a server. Thanks.
But, now I want creat a update OTA app for other AOSP ( andoird L) and using that server which I have just creat.
Can I using CMupdater for it? Can you give me some suggest?

---

Hi, need help to understand!
First of all, i want to say Thank You! For this great job!
I installed server, activated mod rewrite in apache2, apt-get install memcached, made chown for all files to www-data in CyanogenModOTA directory
created directories builds/buildprop, builds/changelog
created file romname.txt in changelog
copied file rom archive to build/full directory
and when i open in browser my ota site: http ota.mydomain.com i see the dir listing as described above
when i try to open url http ota.mydomain.com/api - i see the 404 error.
what exactly i have to do as the next step?
is any json file missed in web root directory (in same place as index.php) ?
and I leave index.php almost unchanged (as in the repository) - just changed string: ->setConfig( 'basePath', '/' )
Thank You once again!

---

vvzar said:
First of all, i want to say Thank You! For this great job!
I installed server, activated mod rewrite in apache2, apt-get install memcached, made chown for all files to www-data in CyanogenModOTA directory
created directories builds/buildprop, builds/changelog
created file romname.txt in changelog
copied file rom archive to build/full directory
and when i open in browser my ota site: http ota.mydomain.com i see the dir listing as described above
when i try to open url http ota.mydomain.com/api - i see the 404 error.
what exactly i have to do as the next step?
is any json file missed in web root directory (in same place as index.php) ?
and I leave index.php almost unchanged (as in the repository) - just changed string: ->setConfig( 'basePath', '/' )
Thank You once again!
Click to expand...
Click to collapse
Are you sure that modrewrite is working? Can you share a working URL?
lingak said:
I have just creat susscess a server. Thanks.
But, now I want creat a update OTA app for other AOSP ( andoird L) and using that server which I have just creat.
Can I using CMupdater for it? Can you give me some suggest?
Click to expand...
Click to collapse
Honestly I don't know, we have to check if OTA app is the same in Lollipop and works of course the same. If so, we're already safe and yes it can work out of the box. If not, we have to fix it. If you already have a working example, feel free to do a pull request

---

JulianXhokaxhiu said:
Are you sure that modrewrite is working? Can you share a working URL?
[email protected]:/home/user# a2enmod rewrite
Module rewrite already enabled
[email protected]:/home/user#
.htaccess:
[email protected]:/home/user# cat /var/www/html/CyanogenModOTA/.htaccess
RewriteEngine On
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)$ index.php [QSA,L]
[email protected]:/home/user#
Is any other rules i have to check?
What about statement <Directory /var/www/html/CyanogenModOTA> ?
Is enouph AllowOverride All and Allow from all ?
test url : http ota.smylink.org
Click to expand...
Click to collapse

---

vvzar said:
JulianXhokaxhiu said:
Are you sure that modrewrite is working? Can you share a working URL?
[email protected]:/home/user# a2enmod rewrite
Module rewrite already enabled
[email protected]:/home/user#
.htaccess:
[email protected]:/home/user# cat /var/www/html/CyanogenModOTA/.htaccess
RewriteEngine On
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)$ index.php [QSA,L]
[email protected]:/home/user#
Is any other rules i have to check?
What about statement <Directory /var/www/html/CyanogenModOTA> ?
Is enouph AllowOverride All and Allow from all ?
test url : http ota.smylink.org
Click to expand...
Click to collapse
Technically it should be enough but the order of where you placed it is important too.
Anyway, going to your URL is just making me think that the PHP code is running well (the redirect to builds folder is triggered by CMOTA Rest Code). You're just missing the htaccess rules.
I'm quite sure you just have to figure out your own server setup to understand if mod_rewrite is properly working (allowing htaccess files to be read and parsed).
Click to expand...
Click to collapse