[Q] Building script help - Android Q&A, Help & Troubleshooting

Hi guys...
I'm a translator for MIUI, so i edit apks a lot.
To make things easier, i made this small tool, to help me automate things.. Something like Android Utility and APK Multitool..
But, i'm no programmer, i just got a lot of help from various places, ending up with this tool.
It's far from perfect, needs a lot of improvement and here's where xda comes in
The script is running more or less fine as it is, but it has some (serious?) issues i can't figure out how to fix..
It's a little tricky to explain, but here goes..
First, i press c to clean everything, the operation completes fine.
I press e. to extract apks from a rom zip, operation completes.
Then i install frameworks, operation completes..
BUT, if i now press 2 to decompile all, i get an error:
Code:
/home/dan/buildtool/functions.sh: line 44: no match: *jar
Invalid choice
#?
Thing is, if i quit the tool and restarts it, press2, then it runs fine... ( the *.jar error is expected, it's the "invalid choice" which is interesting..)
So, it has to be something in the menu function, in a loop somewhere, i don't know..
I was hoping someone could run through the script and perhaps catch the error.. I'm really hoping for a simple answer
Other than that, i could use some good inputs about how to improve the script, add functionality and develop it in general...
The script is on github:
https://github.com/1982Strand/buildtool

---

I don't think you have a looping problem.
Code:
[COLOR=Gray]1067[/COLOR] [[ -z $zip ]] && echo "Invalid choice" && continue
Looks like you explicitly want it to echo "Invalid choice"
I think the problem lies in line #53
Code:
for file in *.apk *jar; do
...you will receive an error if *jar doesn't exist.
You may want to split it up with an if/then...for, for both *.apk and *jar like so:
Code:
if [ -f *.apk ]; then
for file in *.apk; do
...
done
fi
if [ -f *jar ]; then
for file in *jar; do
...
done
fi
[Edit:] Also, you should be aware of this if you aren't already....You can debug your bash scripts with the -vx switch in your shabang statement like so:
Code:
#!/bin/bash -vx

---

okay, i tried changing line #53 as you said, but now i get another error:
Code:
/home/dan/buildtool/functions.sh: line 53: [: too many arguments
/home/dan/buildtool/functions.sh: line 64: no match: *jar
Invalid choice
#?
And with the -vx set in the shebang:
Code:
/home/dan/buildtool/functions.sh: line 53: [: too many arguments
/home/dan/buildtool/functions.sh: line 64: no match: *.jar
++ [[ -z '' ]]
++ echo 'Invalid choice'
Invalid choice
++ continue
#?
Before i changed the code, it was normal that i threw the error with the *.jars not found, but it's supposed to continue anyway..
As i said, if i quit the tool when the error came up, start the tool again and press 2, it works. (it still gives me the error with the *.jar not found, but that was normal)
I don't understand why it takes me back to the prompt as if i were about to extract the apks??
It seems to me it's because i'm still "in" the "e. extract apks from zip" - function...?? I must admit,i don't fully understand the code in that function, so i'm having a hard time troubleshooting it..

---

1982Strand said:
okay, i tried changing line #53 as you said, but now i get another error:
Code:
/home/dan/buildtool/functions.sh: line 53: [: too many arguments
/home/dan/buildtool/functions.sh: line 64: no match: *jar
Invalid choice
#?
And with the -vx set in the shebang:
Code:
/home/dan/buildtool/functions.sh: line 53: [: too many arguments
/home/dan/buildtool/functions.sh: line 64: no match: *.jar
++ [[ -z '' ]]
++ echo 'Invalid choice'
Invalid choice
++ continue
#?
Before i changed the code, it was normal that i threw the error with the *.jars not found, but it's supposed to continue anyway..
As i said, if i quit the tool when the error came up, start the tool again and press 2, it works. (it still gives me the error with the *.jar not found, but that was normal)
I don't understand why it takes me back to the prompt as if i were about to extract the apks??
It seems to me it's because i'm still "in" the "e. extract apks from zip" - function...?? I must admit,i don't fully understand the code in that function, so i'm having a hard time troubleshooting it..
Click to expand...
Click to collapse
Sorry, I forgot using an 'if' statement in that way would produce the "too many arguments" error. This is from '*.apk' having more than one match, so this is what I came up with:
Code:
cd $IN
if [ "$(ls -1 | grep '.\+\.apk$' | wc -l)" -gt 0 ]; then #if there are more than 0 results of *.apk...
for file in *.apk ; do
echo "Decompiling $file" 2>&1 | tee -a $LOG/decompile_log.txt
apktool -q d -f $file $DEC/$file
done
cp -f $HJEM/sort.py $DEC
cd $DEC
python sort.py
rm -r sort.py
fi
if [ "$(ls -1 | grep '.\+\jar$' | wc -l)" -gt 0 ]; then #if there are more than 0 results of *jar...
for file in *jar; do
echo "Decompiling $file" 2>&1 | tee -a $LOG/decompile_log.txt
apktool -q d -f $file $DEC/$file
done
fi
After running options 'e' & 'c', then running option 2, there is no error and the script runs as it should (That is, assuming you chose option 2 of 'c' - "Clean all but apks in apk_in folder". Chosing option 1 of 'c' will obviously result in an error because no .apk or jar files will exist to decompile).
The reason you keep getting the "Invalid choice" error is because you're explicitly asking for it.
With the line #1067 mention earlier and others similar to it like the example below:
Code:
[COLOR=Gray]914[/COLOR] [[ -z $file ]] && echo "Invalid choice" && continue
When the variable string for '$file' has a zero length, as would be the case if '*jar' doesn't exist, the script will echo "Invalid choice". My example above ensures that the '$file' variable string will not have a zero length.
My suggestion would be to change "Invalid choice" to something more specific to the function for which it is being used, that way you can get a better idea of the source of your error.
As far as why you would get that error only after choosing options 'e' & 'c' and not after restarting the script, I couldn't say for sure without digging into it a little more, but at least this fixes your original problem.
I hope that helps.

---

Wow! That did the trick for option 2!!
But then it returns when i continue and get to option 4 "Fix sources":
Code:
[--- Fix MIUI sources ---]
...Fixing framework-miui-res.apk
patching file /home/dan/buildtool/apk_in/decompiled/framework-miui-res.apk/apktool.yml
/home/dan/buildtool/functions.sh: line 132: no match: *.rej
Invalid choice
#?
The point here is, that sometimes the patching fails and patch will generate some files (*.rej and *.orig) that needs to be deleted. But often, like most of the time really, the patching succeeds, so these files are not generated and my simple "remove" commands fail, obviously..
So, i'm guessing here that i get this zero-length issue and my script returns me to this code...
Well, this brings us back to
Code:
[[ -z $file ]] && echo "Invalid choice" && continue
From the "e" option..
This seems to be it.. I'm not entirely sure about what the code exactly means, except the "invalid choice" and continue..
The point with the option "e. Extract apks from zip", is that the user gets a list of zip files contained in the source_rom folder, then choose one.
Then a set of apks (defined in translation_list.txt) must be extracted to apk_in.
The function should simply give the options to choose a zip, x to return to the main menu, or write "invalid choice" if wrong key is entered...
But something seems broken in code, i just can't figure out what and where...
Btw, the code itself is from stackoverflow.com, so i didn't write it myself like that, i just used it for my script as it seemed to do what i needed, but i guess it needs some adjustment still

---

Okay, in the previous example, you had a 'for' loop that was written like this...
Code:
for file in *.apk *jar; do
this
and that
done
...which is saying, For every file (one at a time) in the current directory that matches the patterns *.apk and *jar, assign that filename to the variable '$file', then do..."this and that" while plugging in the value of '$file' for that particular iteration of the loop to the set of commands represented by "this and that". If for some reason, say, no files match the pattern *jar, then for each iteration of the loop regarding that pattern, $file will be equal to ' ' instead of something like 'filename.jar'. That is a variable string length of 0.
Code:
[[ -z $file ]] && echo "Invalid choice" && continue
...what that is, is a test to see if the string (or in this case filename) represented by '$file' has a zero length as with the example above where there were no jar files to assign to the variable '$file'. Similarly, it would most likely be the case with option 4 when there is no match for '*.rej' & '*.orig'.
In my example:
Code:
if [ "$(ls -1 | grep '.\+\.apk$' | wc -l)" -gt 0 ]; then
...I'm testing to see if the output of the command 'ls -1 | grep '.\+\.apk$' | wc -l' is greater than 0, before continuing with the 'for' loop.
The 'ls -1'command lists the contents of the current directory to one column, instead of the typical two or more. The 'wc -l' counts the number of lines in the resulting output. And grep '.\+\.apk$' is a regular expression that makes sure the resulting output only contains filenames that end in '.apk'. So if there are files that end in .apk, then the output of the command 'ls -1 | grep '.\+\.apk$' | wc -l' would be greater than 0, and the same would hold true for jar files. I'm sure there's a more elegant way of doing it, but it works.
Side note: Regular Expressions are powerful pattern matching tools that you definitely need to learn if you want to get the most our of your shell scripts. Google "regexp" or "Bash regexp" to learn more. It can be very confusing to understand at first, but once you get the hang of it, it is really pretty easy.
Anyway, getting back on track...
After running option 2 to decompile, then running option 4 to fix MIUI sources, everything runs fine...even with, or without '*.rej' & '*.orig'. I'll use the following debug as an example:
Code:
+ echo '...Fixing framework-miui-res.apk'
...Fixing framework-miui-res.apk
+ echo ''
+ patch -i /home/soup/buildtool/src_fix/framework-miui-res/apktool.diff /home/soup/buildtool/apk_in/decompiled/framework-miui-res.apk/apktool.yml
patching file /home/soup/buildtool/apk_in/decompiled/framework-miui-res.apk/apktool.yml
+ cd /home/soup/buildtool/apk_in/decompiled/framework-miui-res.apk/
[COLOR=Red]# notice there are no files that match '*.rej' or '*.orig' [/COLOR]
+ rm -f -r '*.rej'
+ rm -f -r '*.orig'
[COLOR=Red]# and there are no errors as a result of it[/COLOR]
+ echo ''
+ echo '...Fixing MiuiCompass.apk'
...Fixing MiuiCompass.apk
+ echo ''
+ patch -i /home/soup/buildtool/src_fix/MiuiCompass/apktool.diff /home/soup/buildtool/apk_in/decompiled/MiuiCompass.apk/apktool.yml
patching file /home/soup/buildtool/apk_in/decompiled/MiuiCompass.apk/apktool.yml
Hunk #1 FAILED at 4.
1 out of 1 hunk FAILED -- saving rejects to file /home/soup/buildtool/apk_in/decompiled/MiuiCompass.apk/apktool.yml.rej
[COLOR=Red]# here, there is now a match available for 'apktool.yml.rej' but not 'apktool.yml.orig'[/COLOR]
+ cd /home/soup/buildtool/apk_in/decompiled/MiuiCompass.apk/
+ rm -f -r apktool.yml.rej
+ rm -f -r apktool.yml.orig
[COLOR=Red]# still no error[/COLOR]
+ echo ''
...so I wouldn't know what to tell you without being able to recreate it on my end.
It may be helpful to give your variables $file and $zip in those functions a non-zero value after they are run to make sure there isn't any zero length hangover from a previous option, like so...
Code:
pull () {
shopt -s failglob
echo "[--- Choose rom zip to extract from, or x to exit ---]"
echo ""
echo ""
select zip in $SRC/*.zip
do
[[ $REPLY == x ]] && . $HJEM/build
[[ -z $zip ]] && echo "Invalid choice" && continue
echo
for apk in $(<$HJEM/translation_list.txt); do
unzip -j -o -q $zip system/app/$apk -d $IN 2&>1 > /dev/null;
done
unzip -j -o -q $zip system/framework/framework-res.apk -d $IN 2&>1 > /dev/null;
unzip -j -o -q $zip system/framework/framework-miui-res.apk -d $IN 2&>1 > /dev/null;
done
zip=dummy [COLOR=Red]<-- after the script is run, assign the string 'dummy' to $zip[/COLOR]
}

---

soupmagnet said:
In my example:
Code:
if [ "$(ls -1 | grep '.\+\.apk$' | wc -l)" -gt 0 ]; then
...I'm testing to see if the output of the command 'ls -1 | grep '.\+\.apk$' | wc -l' is greater than 0, before continuing with the 'for' loop.
The 'ls -1'command lists the contents of the current directory to one column, instead of the typical two or more. The 'wc -l' counts the number of lines in the resulting output. And grep '.\+\.apk$' is a regular expression that makes sure the resulting output only contains filenames that end in '.apk'. So if there are files that end in .apk, then the output of the command 'ls -1 | grep '.\+\.apk$' | wc -l' would be greater than 0, and the same would hold true for jar files. I'm sure there's a more elegant way of doing it, but it works.
Side note: Regular Expressions are powerful pattern matching tools that you definitely need to learn if you want to get the most our of your shell scripts. Google "regexp" or "Bash regexp" to learn more. It can be very confusing to understand at first, but once you get the hang of it, it is really pretty easy.
Click to expand...
Click to collapse
Really good stuff! Learning a lot from this, thanks! I'll dig into regular expressions right away
Anyways, i think i got around the missing *.rej and *.orig by approaching the operation with SED instead of PATCH..
First of all, it makes my code shorter and i don't need an external .diff file for the operation to succeed. (Given that i write the SED code correctly of course..)
It may be helpful to give your variables $file and $zip in those functions a non-zero value after they are run to make sure there isn't any zero length hangover from a previous option, like so...
Code:
pull () {
shopt -s failglob
echo "[--- Choose rom zip to extract from, or x to exit ---]"
echo ""
echo ""
select zip in $SRC/*.zip
do
[[ $REPLY == x ]] && . $HJEM/build
[[ -z $zip ]] && echo "Invalid choice" && continue
echo
for apk in $(<$HJEM/translation_list.txt); do
unzip -j -o -q $zip system/app/$apk -d $IN 2&>1 > /dev/null;
done
unzip -j -o -q $zip system/framework/framework-res.apk -d $IN 2&>1 > /dev/null;
unzip -j -o -q $zip system/framework/framework-miui-res.apk -d $IN 2&>1 > /dev/null;
done
zip=dummy [COLOR=Red]<-- after the script is run, assign the string 'dummy' to $zip[/COLOR]
}
Click to expand...
Click to collapse
Added the code
For now, the script runs through all options fine without halting. Great! But i still need to test it more thoroughly.
Now, i will look into refining the script. Especially the "5. mods" and "10. build flashable zip".
Here's how i'd like it to operate:
When option 4 is processed, i'd like to be able to add some modding to the files. I think it's better to do this before recompiling (option 6/12) because if an apk needs to be edited, it's already decompiled.
For the 3way reboot, it needs to modify some jars. I'd like the user to choose which zip from the source_roms folder to work with and extract the version number the zip filename. (The filename will ALWAYS contain a version number..) So that when the jars are processed, the output files will be placed in a folder with the version number (like /out/"version") This is because, when i want to build my flashable zip, i want the user to input which version number to build it for and then it would pull whatever mods are made for this version number. (Because the entire ROM would probably break if those version numbers don't match)
For the crt-off effect, it should do the same, but it has to check wether a jar file is already existing in /out/"version" and modify that one if it is. (Both mods need to modify the same file)
Right now, i have extra options for OFFICIAL roms. The mods are exactly the same, only the file naming in the function are different. I'd like to eliminate those options, by having the user choose what file to process, like i explain in the above..
Guess that gets a little complicated, hope you get what i mean.. It'll take some time to re-write my functions and the code, but eventually, i'll get there!

---

Ok, so I got it working, with the creation of the folder from the filename. Cool, one step further, I'll continue development tonight or tomorrow

---

Ok, next problem
In the following function, i want the script to check, if any apks exist in the folder. If yes, present the menu to choose which one to decompile. If no, display an error message and return to the main menu.
But something is up with the LS command, no matter if there are files or not in the folder, it returns the message "no files found"..
Having trouble figuring this one out..
Code:
decompile_single () {
shopt -s failglob
echo "[--- Choose apk number, or x to exit ---]"
echo ""
echo ""
cd $IN
if [ "$(ls -A $IN)" ]; then
echo ""
echo "No files found.."
echo ""
else
select file in *.apk
do
cat /dev/null > $LOG/decompile_log.txt
[[ $REPLY == x ]] && . $HJEM/build
[[ -z $file ]] && echo "Invalid choice for single decompiling" && continue
echo
echo "Decompiling $file" 2>&1 | tee -a $LOG/decompile_log.txt
apktool d -f "$file" $DEC/$file
cp -f $HJEM/sort.py $DEC/$file
python $DEC/$file/sort.py
rm -r $DEC/$file/sort.py
break
done
fi
}
Any ideas?

---

Okay...
The output of the command "ls -A $IN" exits with 0 if successful, otherwise it exits with 1. A good way to look at 'if' constructs is...(if "0" (goto)-> then.....if "anything else" (goto)-> else). If you're unsure of what the exit status of a command is, you can enter it in the terminal while piping it into the "echo $?" command. "$?" is a bash variable that represents the exit code of the previous command only.
For your example, you can test the exit status of that command like so...
Code:
ls -A ~/buildtool/apk_in | echo $?
Since the exit status is 0, then your output will be "No files found.." But here's where it gets tricky...The exit status of the 'ls' command will always be 0 unless the directory just cannot be accessed, which is why you will always get the same output..."No files found..". You can find out more about the exit status of a command by visiting its man page (man ls).
To get around this, you need to write the command in such a way that will give you an exit status of anything other than 0 if the condition is not met. Since you only want to check for the existence of ".apk" files you could expand on the command using a regular expression and the 'wc' command, like with my previous example...
Code:
if [ "$(ls -A $IN | grep '.\+\.apk$' | wc -l)" -eq 0 ]; then
echo ""
echo 'No ".apk" files found..'
echo ""
else
...

---

soupmagnet said:
Okay...
The output of the command "ls -A $IN" exits with 0 if successful, otherwise it exits with 1. A good way to look at 'if' constructs is...(if "0" (goto)-> then.....if "anything else" (goto)-> else). If you're unsure of what the exit status of a command is, you can enter it in the terminal while piping it into the "echo $?" command. "$?" is a bash variable that represents the exit code of the previous command only.
For your example, you can test the exit status of that command like so...
Code:
ls -A ~/buildtool/apk_in | echo $?
Since the exit status is 0, then your output will be "No files found.." But here's where it gets tricky...The exit status of the 'ls' command will always be 0 unless the directory just cannot be accessed, which is why you will always get the same output..."No files found..". You can find out more about the exit status of a command by visiting its man page (man ls).
To get around this, you need to write the command in such a way that will give you an exit status of anything other than 0 if the condition is not met. Since you only want to check for the existence of ".apk" files you could expand on the command using a regular expression and the 'wc' command, like with my previous example...
Code:
if [ "$(ls -A $IN | grep '.\+\.apk$' | wc -l)" -eq 0 ]; then
echo ""
echo 'No ".apk" files found..'
echo ""
else
...
Click to expand...
Click to collapse
Yes, thankyou!! Again, learning new stuff..
I just added your earlier code to this function.. Of course, it works like a charm! Facepalm on me! Hehe!
Been reading page after page about regular erxpressions, tests and all kinds of commands this weekend, i kinda stares blind at my code sometimes, haha!

---

1982Strand said:
Yes, thankyou!! Again, learning new stuff..
I just added your earlier code to this function.. Of course, it works like a charm! Facepalm on me! Hehe!
Been reading page after page about regular erxpressions, tests and all kinds of commands this weekend, i kinda stares blind at my code sometimes, haha!
Click to expand...
Click to collapse
I would say, the things you need to be comfortable with are (in order of importance IMO)...
man pages
exit statuses
debugging
regular expressions
pipes
data manipulation
loops
conditions
everything else

---

soupmagnet said:
I would say, the things you need to be comfortable with are (in order of importance IMO)...
man pages
exit statuses
debugging
regular expressions
pipes
data manipulation
loops
conditions
everything else
Click to expand...
Click to collapse
Got some more reading ahead of me
Anyways, i think the script is pretty good now, it suits my needs so far and most of the errors are taken care of..
I can always improve the code, so i'll probably continue developing on this.. Also because it's not perfect at all and still got some flaws here and there...