[Q] Extracting FB contacts from Android's contacts database. - Android Q&A, Help & Troubleshooting

I found this script to extract Facebooks contact info from a Android phone but even though it mostly works and I get all emails and phone numbers the names get left out and I was wondering if someone knew how to fix this script.
I assume it is just a database change since the script was made.
This is all to make it easier to find friends on Google+ since Facebook tries to stop anyone from getting this info.
Code:
#! /bin/sh
###########################################
#
# Android DB contacts exporter (proof-of-concept)
# 2011-01-06
#
# Extracts "certain contacts" from the Android contacts database to a CSV file
# for import into other contact list tools.
#
#
# == Usage ==
#
# 1) Copy your Android contacts database to your computer
# adb pull /data/data/com.android.providers.contacts/databases/contacts2.db ./
# 2) Run this file from the same directory as contacts2.db
# ./fb-extract.sh
# 3) Import csv to an address book of your choice
#
# == Notice ==
#
# This is a proof of concept script. Use at your own risk and check
# with laws and terms of service regarding data usage.
#
###########################################
# In and out files
CONTACTSDB="./contacts2.db"
OUTFILE="./contacts_fb.csv"
# Write table header
echo "First Name,Last Name,E-mail Address,Mobile Phone,Home Phone" >> $OUTFILE
# Get the raw IDs of all facebook contacts
RAWID_LIST=`sqlite3 $CONTACTSDB "SELECT _id from raw_contacts WHERE account_type='com.facebook.auth.login'"`
# Loop through the IDs and extract necessary data
for RAWID in $RAWID_LIST; do
# Get the first and last names
NAME=`sqlite3 $CONTACTSDB "SELECT data2, data3 FROM data WHERE raw_contact_id=$RAWID AND mimetype_id=7"`
FNAME=`echo $NAME | awk '{split($0,a,"|"); print a[1]}'`
LNAME=`echo $NAME | awk '{split($0,a,"|"); print a[2]}'`
# Get the email address
EMAIL=`sqlite3 $CONTACTSDB "SELECT data1 FROM data WHERE raw_contact_id=$RAWID AND mimetype_id=1"`
# Get the phone numbers (mobile & other). Convert international Japanese # to local using `sed`
MPHONE=`sqlite3 $CONTACTSDB "SELECT data1 FROM data WHERE raw_contact_id=$RAWID AND mimetype_id=5 AND data2=2" | sed 's/^81\([0-9]\{7,\}\)/0\1/'`
OPHONE=`sqlite3 $CONTACTSDB "SELECT data1 FROM data WHERE raw_contact_id=$RAWID AND mimetype_id=5 AND data2=7" | sed 's/^81\([0-9]\{7,\}\)/0\1/'`
# If at least one info field was set then write the csv line
if [ -n "$EMAIL" ] || [ -n "$MPHONE" ] || [ -n "$OPHONE" ]; then
echo "$FNAME,$LNAME,$EMAIL,$MPHONE,$OPHONE" >> $OUTFILE
fi
done
Original source of script
http://webcache.googleusercontent.c...c+contacts+sqlite&hl=en&client=ubuntu&strip=1

A simple change
You have to change the line beginning for NAME: at the there should be
mimetype_id=6

Thank you that worked.

Please use the Q&A Forum for questions Thanks
Moving to Q&A

Related

[FAQ][A] About Contacts Import/Export, Backup/Restore and Data/Sync

All About Contacts Import/Export, Backup/Restore and Data/Sync
Last Change: 2011-12-29
Content:
Details about how to backup and restore Contacts on the
Samsung Galaxy S I & II, including data storage info.
Summary:
Backing up and restoring your contacts on the latest Android OS
based phones is a nightmare as long as you do not want to use
automatic sync via Google or Outlook Express. In fact without
paying money for some specialized Application to do it properly
for you, it is a mess. Here I feel Samsung have really failed.
Why there isn't already a built-in simple-to-use contact backup
functionality, that doesn't depend on Google, Kies, Bluetooth
or other net-based service, is way beyond me.
Anyway, here is my attempt to remedy this situation and the
various options you have to backup and restore all your contacts
without:
a) using Google, Microsoft, Kies or other wireless protocol and
b) paying for specialized applications or software.
To simplify this exercise (a lot) we will assume that we would be
satisfied with backup and restore of only the most essential information.
This means (for me):
- first_name
- last_name
- primary_phone
- primary_email
So let's see how far we can get...
NOTE:
a) All of this tutorial assumes you have a rooted phone and
that you are operating as the root user (su)!
b) AOS phones doesn't generally come with the SQLite3
command-line binary, so this have to be installed manually!
Keywords:
Contacts, Restore, CSV, vCard, SQL,
Backup, Data, Sync, SQLite3, Import, Export, VCF
​Content​ - Introduction
- Remounting /system as Read/Write
- Installing the SQLite Command-Line Interpreter
- SQLite3 Install Script
- Contacts Database & Storage-Location:
- Contacts DB Location List
- Direct Manipulation of the Contacts Database
- Other Contact Import / Export Applications
- Help Needed!​Introduction​Essentially there are only 4 different ways to backup your Android contacts.
1. By using a backup application like Kies or 3rd party one from Android Market.
2. By syncing your contacts to your Google account.
3. By copying the entire "Contacts" database to your PC.
4. By exporting only the relevant data directly on your phone, using SQLite.
In this post I will only talk about the last 2 options, with focus on (4) as it applies to (3) as well.
​Remounting /system as Read/Write​Any manipulation of the system partitions and the root directory require WRITE permission to the file-system. To find out, just use the mount command and look for the "rw" label in the filesystem description of the / and /system partitions. For example:
Code:
# mount
...
rootfs / rootfs ro,relatime 0 0
/dev/block/mmcblk0p9 /system ext4 ro,relatime,barrier=1,data=ordered 0 0
/dev/block/mmcblk0p10 /data ext4 rw,nosuid,nodev,noatime,...,discard 0 0
/dev/block/mmcblk0p4 /mnt/.lfs j4fs rw,relatime 0 0
/dev/block/vold/179:11 /mnt/sdcard vfat rw,dirsync,...,utf8,...,discard 0 0
...
See the "rw" following "/system ext4" and "/data ext4" above?
Yes? Good, you can skip to the next section!
No? Then you need to remount the filesystem as READ/WRITE.
("rw" = "Read AND Write" and "ro" = "Read Only".)
!! However, this may depend on your model AND on how you connected your phone to USB. You can enter the phone directly without using the "file-transfer" mode in which case you may or may not have RW, but if you select to use "file-transfer" (aka. USB-disk mode) you may only get RO. In some cases the /mnt/sdcard is simply never mounted. Try and test! [*** check this! ***]
The way to get RW, is to remap the "physical" device (disk) location to the system partition you are interested in. The tricky part is knowing the physical device name of your system disk which may not always be the same as indicated from the mount command. If not, try to inspect the boot-log messages with the "motd" command, or inspecting the file "/init.rc" for the details of everything that is mounted upon boot. (Boot-log messages are normally enabled in most stock Kernels, but not necessarily in modded ROM's.)
For the SGS-2 which uses the EXT4 filesystem, I had to use:
Code:
mount -o remount,rw -t ext4 /dev/block/mmcblk0p9 /system
For the SGS-1 when using an RFS based AOS, I had to use:
Code:
mount -o remount,rw -t yaffs2 /dev/block/mtdblock3 /system
To undo this you just replace "rw" with "ro" in the statements above. Although this is not necessary, since it will be automatically reverted after next reboot, which is recomended anyway after manually editing any system DB.
​Installing the SQLite Command-Line Interpreter​Most phones doesn't come with the SQLite3 command-line interpreter/interface (CLI) installed, unless they are rooted with a heavily modded ROM. The CLI is needed to manually access the SQLite Contacts database. Thus you need to check that it is not already installed and install it manually, if it is not. The binary (sqlite3) is normally located in either /system/bin or /system/xbin. To check if you have it, try the following command:
Code:
find / -iname '*sqlite*'
If it fails to return the files shown below, you need to install the SQLite3 binary AND possibly the relevant library files. You should already have most of the required library (lib*.so) files, as they are constantly used by the system, so look for them first! The files you need are:
/system/bin/sqlite3
/system/lib/libsqlite.so
/system/lib/libsqlite_jni.so
If the binary is compiled from CyanogenMod sources, you will also need the following:
/system/lib/libncurses.so
In the worst case you can go through the exercise to compile the SQLite3 sources for the MIPS32 chipset by yourself. [http://www.sqlite.org/] Also it's worth to note that some custom ROM's now include the latest compiled version. When you have the binary you just put it on your SD card and copy it to the root file system binary directory. Many ROM's and rooted phones use an additional directory /system/xbin where you can put your own binaries. If this directory (and path variable) is not already present, use:
Code:
cp <path_to_your_sqlite3_binary> /system/bin/sqlite3
chmod 4755 /system/bin/sqlite3
chown root.shell /system/bin/sqlite3
(Note that this version of chown uses a "." in the "user.group" specification.)
Do the same for the library files, if they are missing. You should now be able to run SQLite3:
Code:
# sqlite3 -version
3.6.22
Everything OK? Good!
​SQLite3 Install Script​!!MODIFY FIRST!!
Code:
#!/bin/sh
# squint.sh - sqlite3 install script for rooted Androids
#
#-rwsr-sr-x root shell 22228 2011-11-10 12:53 /system/bin/su
#-rwsr-xr-x root shell 1075144 2011-12-15 16:55 /system/xbin/busybox
#-rw-r--r-- root root 322120 2011-09-14 14:45 /system/lib/libsqlite.so
# Go to your "sqlite" folder on your (internal) SD card:
cd /mnt/sdcard/sqlite/
cp sqlite3 /system/xbin/sqlite3
chmod 4755 /system/xbin/sqlite3
chown root.shell /system/xbin/sqlite3
cp ./lib/libncurses.so /system/lib/libncurses.so
chmod 644 /system/lib/libncurses.so
chown root.root /system/lib/libncurses.so
Contacts Database & Storage-Location​Apparently the SQLite3 database used to store the Contacts data is located in different parts on different Android models [?] and different Android OS versions. I have not been able to verify this, as I don't have access to these phones, but from from reading various forum posts for other phones this seem to be the case.
The way you can check is by looking for the raw database file on your system. On the phones I have, the location can be found here:
SGS-1 (2.3.6): /dbdata/databases/com.android.providers.contacts/contacts.db
SGS-2 (2.3.4): /data/data/com.android.providers.contacts/databases/contacts2.db
It may be important to notice that the "com.android.providers.contacts" part of this path, is actually referring to the particular "Contacts.apk" application that is installed in your phone. Therefore this location can differ from other versions of that application and thus also your phone model and OS version. In addition note that the /data and /dbdata or "whatever" partitions may be mounted on a different block device, as noted in the first section above. For example, on my SGS-2, I had /data mounted on /dev/block/mmcblk0p10 .
The more intelligent way is to look for all .db files on you system by doing some shell magic:
Code:
find / -iname '*.db'
But today, almost every application is using a database to save information, so to narrow down the long list, filter your search with "ontact":
Code:
find /system -iname '*.db' |grep ontact
Contacts DB Location List​GT-I9000 (2.3.6): /dbdata/databases/com.android.providers.contacts/contacts.db
GT-I9100 (2.3.4): /data/data/com.android.providers.contacts/databases/contacts2.db
[I'd like to keep this list updated with other models and AOS versions, so please let me know what you have.]
​Direct Manipulation of the Contacts Database​In order to make any changes to any information contained in the database, you need to have write permissions set on the the relevant partition(s) as explained above. Remount as needed.
If you for some crazy reason need to replace/restore your entire contact database with another one. (This is strongly discouraged!) You also need to make sure that the one you replace with have both the same ownership (chown) and permissions (chmod) as the original one.
Next, let's open the contacts database:
Code:
# cd /data/data/com.android.providers.contacts/databases
# sqlite3 contacts2.db
SQLite version 3.6.22
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> .databases
.databases
seq name file
--- --------------- ----------------------------------------------------------
0 main /data/data/com.android.providers.contacts/databases/contacts2.db
sqlite> .tables
.tables
_sync_state settings
_sync_state_metadata speed_dial
accounts status_updates
activities v1_settings
agg_exceptions view_contacts
android_metadata view_contacts_restricted
calls view_data
contact_entities_view view_data_restricted
contact_entities_view_restricted view_groups
contacts view_raw_contacts
data view_raw_contacts_restricted
groups view_v1_contact_methods
mimetypes view_v1_extensions
name_lookup view_v1_group_membership
nickname_lookup view_v1_groups
packages view_v1_organizations
phone_lookup view_v1_people
properties view_v1_phones
raw_contacts view_v1_photos
sqlite> SELECT _id,name, number FROM view_v1_phones;
[...]
To output the result as a CSV table to a file called "my_contacts_db.csv" with the first row containing the (column) headers and bailing out on errors, do this:
Code:
sqlite> .bail ON
sqlite> .headers ON
sqlite> .mode csv
sqlite> .output my_contacts_db.csv
sqlite> SELECT _id,name, number FROM view_v1_phones;
sqlite> .output stdout
sqlite> .quit
As you can see, if we had known the precise query statements we would have already been done with our export. But the Contacts.db tables are rather complex for our purposes of simply extracting the most relevant contact information. I lazily gave up after trying some basic SQL select statements...
[So yeah, please send me a good SQL statement for selecting and printing only the basics as a CSV!]
The AOS java API for using the contacts are MUCH simpler to use than the needed SQLite statements, for doing the extractions/replacements. But then you also need to write the program.
http://www.higherpass.com/Android/Tutorials/Working-With-Android-Contacts/
There are many such programs out there, but very few who does the job well. (Everybody want to make money!) So try some Google-Fu on sites such as:
http://www.sourceforge.com/
http://code.google.com/
http://www.github.com/
http://www.stackexchange.com/
http://grepcode.com/
and use the keywords in the top of this document.
​Other Contact Import / Export Applications​But I'm not a Java Programmer. (What?!!) So instead I had a look at the most simple applications for exporting and importing you contacts. Being reasonably paranoid as a person, I did not trust huge multifunction backup-applications, knowing that they are full of adware, spyware and other "things" we may not like. With these limits I knew that any such application should not be much larger than 100 kilobytes.
I found 4 small applications (and one very large) on the Market that I tested:1. Contacts CSV Export (93 K)
https://market.android.com/details?id=dutchandroid.contacts.export
(contacts.export-1060-1.060.apk, dutchandroid.contacts.export)
2. Contacts Backup Trial (168K)
https://market.android.com/details?id=com.saturnat.android.contactsbackup.trial
(contactsbackup.trial-34-2.9.5.apk, com.saturnat.android.contactsbackup.trial)
3. Import Contacts (56 K)
https://market.android.com/details?id=org.waxworlds.edam.importcontacts
(import-contacts_1.1.apk, org.waxworlds.edam.importcontacts)
4. Offline Contacts Sync (27 K)
https://market.android.com/details?id=com.beforemadness
(offlineContactsSync.apk, com.beforemadness)
5. GO Contacts EX (2.7 M!)
https://market.android.com/details?id=com.jbapps.contactpro​(1) Is a flexible lightweight CSV contacts exporter. You can manually choose CSV delimiters (",;TAB" etc) or choose a standard (Mozilla, Hotmail and Outlook etc.) CSV format.
(2) $$$, Is buggy with a no-sense UI, doesn't catch Email addresses, don't let you choose filename, and contain trackers from Facebook and Mixpanel. Limited trial to 10 contacts and 3 days before expiration.
(3) Only imports vCard (.vcf) files. One-by-one or scan a directory. Ask's what to do (skip,merge,replace) with duplicates.
(4) No-sense UI. Has export function but it is not working. And import is only working partially.
(5) Exports all contacts to a single vCard file, but missed my Emails. Nice merge and batch-delete functionality. Nice interface, and works. However, this app is huge, and god knows what other bloat junk it contains.
​Other Useful Links​[1] http://www.sqlite.org/
[2] http://www.androidadb.com/source/pl...ndroid/contacts/ImportVCardActivity.java.html
[3] http://grepcode.com/project/repository.grepcode.com/java/ext/com.google.android/android/
[4] sed Tutorial: http://www.grymoire.com/Unix/Sed.html
​
Help Needed!​If you have any detailed knowledge about where and how the relevant contact data is stored on your device, please let me know and include details such as:
- Your phone model number
- Your general Android OS version [Settings --> About Phone ]
- Your phone firmware version (PDA / PHONE / CSC) ["dial" --> *#1234#]
- The precise location of the contacts database file
E.g.
GT-I9100, Gingerbread 2.3.4, XWKI4/XXKI1/OXX KI2 (SEB)
/dbdata/databases/com.android.providers.contacts/contacts.db
If you have already been poking around your OS it would also be helpful to know:
- The version and type of database used (SQLite3 etc.)
- The particular DB tables used to obtain:
+ First Name
+ Last Name
+ Phone Number "1"
+ Email "1"
Here I have written "1" to signify the primary number used. I have refrained from using more details as this, as I consider this minimal but sufficient for backing up and restoring my contacts.
​
Some useful SQLite queries​
Code:
sqlite3 -csv contacts2.db "SELECT _id,name,primary_email,number FROM view_v1_phones;"
sqlite3 -csv contacts2.db "SELECT _id,mimetype_id,raw_contact_id,data1 FROM data;"
sqlite3 -csv contacts2.db "SELECT _id,display_name FROM raw_contacts WHERE account_type='vnd.sec.contact.phone';"
sqlite3 -csv contacts2.db "SELECT data1 FROM data WHERE raw_contact_id=112 AND mimetype_id=5;"
sqlite3 -csv -header contacts2.db "SELECT _id,package_id,mimetype_id,raw_contact_id,is_primary,is_super_primary,data_version,data1,data2,data3 FROM data WHERE raw_contact_id=21;"
sqlite3 -csv -header contacts2.db "SELECT _id,package_id,mimetype_id,raw_contact_id,is_primary,is_super_primary,data_version,data1,data2,data3 FROM data WHERE is_primary=0 AND data_version=0;"
#SELECT data1 FROM data WHERE raw_contact_id=140 AND mimetype_id=1;
#SELECT data1 FROM data WHERE raw_contact_id=140 AND mimetype_id=5 AND data2=2;
#SELECT data1 FROM data WHERE raw_contact_id=112 AND mimetype_id=5 AND data2=7;
#SELECT data1 FROM data WHERE raw_contact_id=112 AND mimetype_id=5 AND data2=1;
If this helped you in any way, please hit the THANK YOU button! ​
The Android contacts-database extraction script
This script is based on one that I found for extracting Facebook contacts into a CSV file, here: http://forum.xda-developers.com/showthread.php?t=1170765 I then modified and improved this script to suit my own purposes, but I am very grateful to whoever created it in the first place. (There is no reliable reference to the origin of this.)
However, there are two very useful things I found and included in this script.
1) You can extract contacts from other applications using that same database file, by modifying the "java-path-identifyer" accordingly.
Code:
Specify the application used for organizing the data:
-----------------------------------------------------------
vnd.sec.contact.phone - Default Android phone book
com.facebook.auth.login - Facebook contacts
2) The sed pipe used to replace carrige-returns (new-lines, \n) from any output stream, with a comma ",".
Code:
sed ':a;N;$!ba;s/\n/,/g'
Explanation:
Create and label a register via :a
Append the current and next line to the register via N
If we are before the last line, branch (b) to the created
register (a) with $!ba ( "$!" means not to do it on the
last line, as there should be one final newline).
Use global match (/g) to substitute (s) every newline (\n)
with a comma on the pattern space, which is the contents
of the (a) register. (I.e. The whole input stream.)
Here is the script:
Code:
#!/bin/sh
# contex.sh - Extracts the Name,Phone,Email from the Android Contact Database
#==============================================================================
# Last Change: 2011-12-29
# Name: Android DB Contacts Exporter
#
# Description: Extracts contacts from the Android contacts database
# to a CSV file for import into other contact list tools.
#
# Usage: ./contex.sh <output_filename> &
#
# How to use:
#
# 1. Copy your Android Contacts Database to your computer:
# adb pull /data/data/com.android.providers.contacts/databases/contacts2.db ./
# 2. Run this file from the same directory as contacts2.db
# ./contex.sh output_filename.csv
#
# System Requirements: sh, sqlite3, sed
#
# Additional Notes:
#
# 1. This script is very slow, please be patient!
#
# 2. The sed sequence to replace the end on line (\n) with a comma (,)
# from a list, but avoiding \n on the last line is:
#
# sed ':a;N;$!ba;s/\n/,/g'
# --- OR ---
# sed '{:q;N;s/\n//g;t q}'
#
# 3. Only numbers that are of the types: HOME, MOBILE, WORK and OTHER
# get extracted in this script. These types are specified by the
# "data2" field in the "data" table.
#
#==============================================================================
echo; echo "Android contacts backup script started with PID: $$ "; echo;
DATE=`date +%G%m%d`
#----------------------------------------------------------
# Input and Output files:
#----------------------------------------------------------
CONTACTSDB="./contacts2.db"
if [ ! -f "$CONTACTSDB" ]
then
echo "Contacts Database file: $CONTACTSDB not found. Exiting."; echo
exit 1
fi
OUTFILE=$1
if [ -z "$OUTFILE" ]
then
OUTFILE="contex_backup_${DATE}.csv"
fi
echo "Contacts backup file: $OUTFILE "; echo
if [ -f "$OUTFILE" ]
then
echo "Contacts Backup file: $OUTFILE already exist. Exiting."; echo
exit 1
fi
#----------------------------------------------------------
# Write the Table Header:
#----------------------------------------------------------
echo "First Name,Last Name,E-mail,Mobile Phone,Home Phone,Work Phone,Other Phone" >> $OUTFILE
#----------------------------------------------------------
# Setting up SQLite command and mode:
#----------------------------------------------------------
#CMD="sqlite3 -bail -noheader -csv"
CMD="sqlite3";
#----------------------------------------------------------
# Get the raw IDs of all contacts:
#----------------------------------------------------------
RAWID_LIST=`$CMD $CONTACTSDB "SELECT _id from raw_contacts WHERE account_type='vnd.sec.contact.phone'"`
#echo "$RAWID_LIST"
#----------------------------------------------------------
# Loop through the IDs and extract all necessary data
#----------------------------------------------------------
for RAWID in $RAWID_LIST; do
# Get the First_Name and Last_Name in one query:
# NOTE: Need to remove occasional CTRL-M's with sed.
NAME=`$CMD -csv $CONTACTSDB "SELECT data2, data3 FROM data WHERE raw_contact_id=$RAWID AND mimetype_id=6;" | sed ':a;N;$!ba;s/\n//g'`
# Get the Email address:
# NOTE: This will simply append multiple email adresses to one long string, with ";" separator!
EMAIL=`$CMD $CONTACTSDB "SELECT data1 FROM data WHERE raw_contact_id=$RAWID AND mimetype_id=1;" | sed ':a;N;$!ba;s/\n/;/g'`
# Here we get the "Home", "Mobile", Work" and "Other" (labeled) phone numbers.
# NOTE: Each of these may return multiple responses, so we need to convert new-lines (\n) to ",".
HPHONE=`$CMD $CONTACTSDB "SELECT data1 FROM data WHERE raw_contact_id=$RAWID AND mimetype_id=5 AND data2=1;" | sed ':a;N;$!ba;s/\n/,/g'`;
MPHONE=`$CMD $CONTACTSDB "SELECT data1 FROM data WHERE raw_contact_id=$RAWID AND mimetype_id=5 AND data2=2;" | sed ':a;N;$!ba;s/\n/,/g'`;
WPHONE=`$CMD $CONTACTSDB "SELECT data1 FROM data WHERE raw_contact_id=$RAWID AND mimetype_id=5 AND data2=3;" | sed ':a;N;$!ba;s/\n/,/g'`;
OPHONE=`$CMD $CONTACTSDB "SELECT data1 FROM data WHERE raw_contact_id=$RAWID AND mimetype_id=5 AND data2=7;" | sed ':a;N;$!ba;s/\n/,/g'`;
PHONES="$MPHONE,$HPHONE,$WPHONE,$OPHONE";
# If at least one main field is found then write the CSV line
if [ -n "$NAME" ] || [ -n "$EMAIL" ] || [ -n "$PHONES" ]; then
#echo "ID: $RAWID"
echo -n "$NAME,$EMAIL,$PHONES" >> $OUTFILE
echo >> $OUTFILE
fi
done
echo 'Done.'
NOW YOU CAN HIT THE THANK YOU BUTTON!​
reserved for me 2!
Thanks for this write up. I found my stuck after upgrading a friends phone from 2.2 firmware to CM10 with 4.1 firmware. We made a full backup with TitaniumBackup but it would import the contacts properly. I pulled the contacts2.db from the backup file and used your script to get the data out.
I did have to make some changes to the mimetypes in the SQLs but apart from that everything worked perfectly. I think the mimetypes may have changed between android versions.
The modified portion of the script:
Code:
# Get the First_Name and Last_Name in one query:
# NOTE: Need to remove occasional CTRL-M's with sed.
NAME=`$CMD -csv $CONTACTSDB "SELECT data2, data3 FROM data WHERE raw_contact_id=$RAWID AND mimetype_id=5;" | sed ':a;N;$!ba;s/\n//g'`
# Get the Email address:
# NOTE: This will simply append multiple email adresses to one long string, with ";" separator!
EMAIL=`$CMD $CONTACTSDB "SELECT data1 FROM data WHERE raw_contact_id=$RAWID AND mimetype_id=1;" | sed ':a;N;$!ba;s/\n/;/g'`
# Here we get the "Home", "Mobile", Work" and "Other" (labeled) phone numbers.
# NOTE: Each of these may return multiple responses, so we need to convert new-lines (\n) to ",".
HPHONE=`$CMD $CONTACTSDB "SELECT data1 FROM data WHERE raw_contact_id=$RAWID AND mimetype_id=6 AND data2=1;" | sed ':a;N;$!ba;s/\n/,/g'`;
MPHONE=`$CMD $CONTACTSDB "SELECT data1 FROM data WHERE raw_contact_id=$RAWID AND mimetype_id=6 AND data2=2;" | sed ':a;N;$!ba;s/\n/,/g'`;
WPHONE=`$CMD $CONTACTSDB "SELECT data1 FROM data WHERE raw_contact_id=$RAWID AND mimetype_id=6 AND data2=3;" | sed ':a;N;$!ba;s/\n/,/g'`;
OPHONE=`$CMD $CONTACTSDB "SELECT data1 FROM data WHERE raw_contact_id=$RAWID AND mimetype_id=6 AND data2=7;" | sed ':a;N;$!ba;s/\n/,/g'`;
PHONES="$MPHONE,$HPHONE,$WPHONE,$OPHONE";
Thanks so much for your post.
I can't thank you enough for this information!
My S2 died on me a month back, and the only backup of my contacts (which are not synced on Gmail) were an old TB backup from April 2012. At least thanks to this knowledge I know what and where to look for!
Thanks a lot
Help request with command
Hello.
I am trying to convert a recovered contacts database from contacts2.db to Google compatible cvs file with sqlite3.exe.
The recovered database came from a Samsung N7000 and it is composed as follow in the table data:
SELECT "_id", "package_id", "mimetype_id", "raw_contact_id", "is_primary", "is_super_primary", "data_version", "data1", "data2", "data3", "data4", "data5", "data6", "data7", "data8", "data9", "data10", "data11", "data12", "data13", "data14", "data15", "data_sync1", "data_sync2", "data_sync3", "data_sync4", "is_read_only" FROM "data";
There are 2 lines for a single contact. mimetype_id 6 is the line of the name in data1; mimetype_id 5 is the line of the number in data1
Is there a command to properly export that database? I have no skills in programming and scripting.
I have to complete an import of a database to a new android phone from a N7000 old contacts2.db :crying:
Just to say that the code
Code:
SELECT "_id", "package_id", "mimetype_id", "raw_contact_id", "is_primary", "is_super_primary", "data_version", "data1", "data2", "data3", "data4", "data5", "data6", "data7", "data8", "data9", "data10", "data11", "data12", "data13", "data14", "data15", "data_sync1", "data_sync2", "data_sync3", "data_sync4", "is_read_only" FROM "data";
gives something like that:
5406,,5,2335,0,0,0,349563erased for privacy,2,
5407,,6,2335,0,0,0,"federico agronomo",federico,agronomo
And so on...
Since I have no skills in developing and scripting, is there anybody how can help me extracting universal VCF from that N7000 sql contacts2.db specific database? I can send it to you by eMail if you write to me in pvt. I'll apprecciated.
I did a lot of tests, but I can not create a proper query or run a proper script.
I was able only to extract the database in a way like the one mentioned. I have quite 1000 contacts :crying:
Thank you in any way for help.
Kind regards.
THREAD CLOSED!
There are now hundreds of contacts & backup Apps on Google Play, so I'm closing this old thread.
But here are a few free ones that I really like:
Contact Backup & Restore Gmail
Super Backup : SMS & Contacts
MCBackup - My Contacts Backup
They should be able to export to CSV, VCF and XML files...
ENJOY!

[Q] Help with ADB or rather a simple command. Im confused!!

Ok, so basically Im trying to extract data from my contacts2.db file.
Ive managed to copy it out of the hidden data folder on my GalaxySII onto my PC HDD. Now i need to run a script to extract the data. The thing is, im baffled how to do this.
Im a n00b when it comes to ADB, but i managed to drop the script and the copied contact2.db file into a test folder on my device, i managed to use the terminal emulator to browse to the directory and run the command but I get: 'Permission Denied' when I run the script. Ive tried changing the permissions on the file via my SII but to no avail.
So, im following the instructions I have via my PC:
== Usage ==
Copy your Android contacts database to your computer:
adb pull /data/data/com.android.providers.contacts/databases/contacts2.db ./
Run this file from the same directory as contacts2.db
./fb-extract.sh
Import csv to an address book of your choice
Now ive got the db, but how do I run the script. Ive tried to load adb emulator but have no idea what im doing there. Ive tried to use the adb command but again, no idea.
Can someone give me a brief, and quick step-by-step way to run this script on my contacts file so I can extract the data I want please.
Thanks muchly....
-=stylus=-
What are you trying to accomplish? The instructions say you should be running the shell script on the PC, not the phone. But you can't just natively run shell scripts in Windows, which I presume is what you're using. You'd need Cygwin or similar assuming you're averse to Linux, and possibly other stuff, depending on what the script is doing.
That fb-script you're trying to run is probably extracting the data with sqlite3. So you need to have the sqlite3 binary available where you are running the script (phone or computer). There are many things which can go wrong, first step would be to paste the content of fb-extract.sh here.
Ah, yes sorry. Im trying to extract all email addresses from my contacts on my phone. Facebook syncs with your Android device and adds all the email addresses registered with your contacts.
The script I have should extract all this data out:
(Not sure if Im allow to post code but)
#! /bin/sh
###########################################
#
# Android DB contacts exporter (proof-of-concept)
# 2011-01-06
#
# Extracts "certain contacts" from the Android contacts database to a CSV file
# for import into other contact list tools.
#
#
# == Usage ==
#
# 1) Copy your Android contacts database to your computer
# adb pull /data/data/com.android.providers.contacts/databases/contacts2.db ./
# 2) Run this file from the same directory as contacts2.db
# ./fb-extract.sh
# 3) Import csv to an address book of your choice
#
# == Notice ==
#
# This is a proof of concept script. Use at your own risk and check
# with laws and terms of service regarding data usage.
#
###########################################
# In and out files
CONTACTSDB="./contacts2.db"
OUTFILE="./contacts_fb.csv"
# Write table header
echo "First Name,Last Name,E-mail Address,Mobile Phone,Home Phone" >> $OUTFILE
# Get the raw IDs of all facebook contacts
RAWID_LIST=`sqlite3 $CONTACTSDB "SELECT _id from raw_contacts WHERE account_type='com.facebook.auth.login'"`
# Loop through the IDs and extract necessary data
for RAWID in $RAWID_LIST; do
# Get the first and last names
NAME=`sqlite3 $CONTACTSDB "SELECT data2, data3 FROM data WHERE raw_contact_id=$RAWID AND mimetype_id=7"`
FNAME=`echo $NAME | awk '{split($0,a,"|"); print a[1]}'`
LNAME=`echo $NAME | awk '{split($0,a,"|"); print a[2]}'`
# Get the email address
EMAIL=`sqlite3 $CONTACTSDB "SELECT data1 FROM data WHERE raw_contact_id=$RAWID AND mimetype_id=1"`
# Get the phone numbers (mobile & other). Convert international Japanese # to local using `sed`
MPHONE=`sqlite3 $CONTACTSDB "SELECT data1 FROM data WHERE raw_contact_id=$RAWID AND mimetype_id=5 AND data2=2" | sed 's/^81\([0-9]\{7,\}\)/0\1/'`
OPHONE=`sqlite3 $CONTACTSDB "SELECT data1 FROM data WHERE raw_contact_id=$RAWID AND mimetype_id=5 AND data2=7" | sed 's/^81\([0-9]\{7,\}\)/0\1/'`
# If at least one info field was set then write the csv line
if [ -n "$EMAIL" ] || [ -n "$MPHONE" ] || [ -n "$OPHONE" ]; then
echo "$FNAME,$LNAME,$EMAIL,$MPHONE,$OPHONE" >> $OUTFILE
fi
done
Now like I said, I have the contacts2.db file, I just am not sure how to go about running this script on it to accomplish my task.
I understand a bit of coding but not much so bear with me.
-=stylus=-
Well, it's obvious you have to run it on a Linux/Android system with /bin/sh and sqlite3 binaries. So, probably better to run it on your phone. But first, you need to have the sqlite3 binary from somewhere, as it's not shipped by default.
Get SuperOneClick and extract it; in the Dependencies folder you'll find the sqlite3 file.
Push it to the phone from your computer (these commands are run in command prompt):
Code:
C:\> adb remount /system
C:\> adb push /path/to/sqlite3 /system/xbin/sqlite3
Check the sqlite3 binary is present in the system path and available:
Code:
C:\> adb shell
[email protected]# which sqlite3
/system/xbin/sqlite3
Now you can run your script. With adb shell, run the script on your phone in the same folder where you have the .db file. Should create a new CSV file which can be pulled to your computer.
Now i just get the error message:
./fb-extract.sh: cannot execute - permission denied
chmod 777 fb-extract.sh
Depending on what folder you're in, you may need to remount r/w too.
And make sure you're not having the two files on a partition mounted with noexec
I wish I was good at this stuff....
Ok, its in a folder:
/sdcard/test
contents of this folder are:
contacts2.db
fb-extract.sh
Ive run the 'chmod 777 fb-extract.sh' command but still permission denied.
Should I put it in a different folder and run it? Is there a better way?
Sorry for my lack of knowledge here but no idea what remount r/w (well other than remount read/write) is or how to do it. How do I know if ive got it on a partition mounted with noexec?
In "adb shell":
Code:
[email protected] # mount | grep sdcard
and post the output. If you see a "noexec" flag, then it doesn't allow you to exec anything from the sdcard. You could try after that
Code:
[email protected] # mount -o remount,exec /sdcard
and try the script again.
I think im losing the will to live guys ha ha ha!!!
Ok the output was:
Code:
a/local/bin:$PATH <
[email protected]:/ $ su
[email protected]:/ # mount | grep sdcard
/dev/block/vold/259:3 on /mnt/sdcard type vfat (rw,dirsync,nosuid,nodev,noexec,noatime,nodiratime,uid=1000,gid=1015,fmask=0002,dmask=0002,allow_utime=0020,codepage=cp437,iocharset=iso8859-1,shortname=mixed,utf8,errors=remount-ro)
tmpfs on /mnt/sdcard/external_sd type tmpfs (rw,dirsync,nosuid,nodev,noexec,noatime,nodiratime,size=0k,mode=755,gid=1000)
tmpfs on /mnt/sdcard/usbStorage type tmpfs (rw,dirsync,nosuid,nodev,noexec,noatime,nodiratime,size=0k,mode=755,gid=1000)
/dev/block/vold/179:25 on /mnt/sdcard/external_sd type vfat (rw,dirsync,nosuid,nodev,noexec,noatime,nodiratime,uid=1000,gid=1023,fmask=0002,dmask=0002,allow_utime=0020,codepage=cp437,iocharset=iso8859-1,shortname=mixed,utf8,errors=remount-ro)
tmpfs on /mnt/sdcard/external_sd/.android_secure type tmpfs (ro,relatime,size=0k,mode=000) [email protected]:/ # mount -o remount,exec /sdcard
mount: can't find /sdcard in /proc/mounts
1|[email protected]:/#
Now im doing all this using terminal emulator on my phone. Thats the same as adb right? Still getting permission denied though.
You would think extracting email addresses from your contacts list was easy!
/mnt/sdcard type vfat (rw,dirsync,nosuid,nodev,noexec
This is what I was saying before.
On your phone with adb:
Code:
[email protected] # mount -o rw,remount /mnt/sdcard
and then you should be able to execute the script.
Well, it's difficult because it wasn't designed to be used that way (although what you are trying to do is quite simple). It would have been easier to sync your contacts with Google and download the CSV from Google Contacts' web interface.
Nope. Still cant do it.. still permission denied.
I cant sync with Google, create a vcard or anything. The facebook emails are protected somehow. If you export your contacts list to any file, it leaves all the facebook emails out!
Agghhhhhhh!
Thanks for all your help though. Much appreciated!!!

[Q] Building script help

Hi guys...
I'm a translator for MIUI, so i edit apks a lot.
To make things easier, i made this small tool, to help me automate things.. Something like Android Utility and APK Multitool..
But, i'm no programmer, i just got a lot of help from various places, ending up with this tool.
It's far from perfect, needs a lot of improvement and here's where xda comes in
The script is running more or less fine as it is, but it has some (serious?) issues i can't figure out how to fix..
It's a little tricky to explain, but here goes..
First, i press c to clean everything, the operation completes fine.
I press e. to extract apks from a rom zip, operation completes.
Then i install frameworks, operation completes..
BUT, if i now press 2 to decompile all, i get an error:
Code:
/home/dan/buildtool/functions.sh: line 44: no match: *jar
Invalid choice
#?
Thing is, if i quit the tool and restarts it, press2, then it runs fine... ( the *.jar error is expected, it's the "invalid choice" which is interesting..)
So, it has to be something in the menu function, in a loop somewhere, i don't know..
I was hoping someone could run through the script and perhaps catch the error.. I'm really hoping for a simple answer
Other than that, i could use some good inputs about how to improve the script, add functionality and develop it in general...
The script is on github:
https://github.com/1982Strand/buildtool
I don't think you have a looping problem.
Code:
[COLOR=Gray]1067[/COLOR] [[ -z $zip ]] && echo "Invalid choice" && continue
Looks like you explicitly want it to echo "Invalid choice"
I think the problem lies in line #53
Code:
for file in *.apk *jar; do
...you will receive an error if *jar doesn't exist.
You may want to split it up with an if/then...for, for both *.apk and *jar like so:
Code:
if [ -f *.apk ]; then
for file in *.apk; do
...
done
fi
if [ -f *jar ]; then
for file in *jar; do
...
done
fi
[Edit:] Also, you should be aware of this if you aren't already....You can debug your bash scripts with the -vx switch in your shabang statement like so:
Code:
#!/bin/bash -vx
okay, i tried changing line #53 as you said, but now i get another error:
Code:
/home/dan/buildtool/functions.sh: line 53: [: too many arguments
/home/dan/buildtool/functions.sh: line 64: no match: *jar
Invalid choice
#?
And with the -vx set in the shebang:
Code:
/home/dan/buildtool/functions.sh: line 53: [: too many arguments
/home/dan/buildtool/functions.sh: line 64: no match: *.jar
++ [[ -z '' ]]
++ echo 'Invalid choice'
Invalid choice
++ continue
#?
Before i changed the code, it was normal that i threw the error with the *.jars not found, but it's supposed to continue anyway..
As i said, if i quit the tool when the error came up, start the tool again and press 2, it works. (it still gives me the error with the *.jar not found, but that was normal)
I don't understand why it takes me back to the prompt as if i were about to extract the apks??
It seems to me it's because i'm still "in" the "e. extract apks from zip" - function...?? I must admit,i don't fully understand the code in that function, so i'm having a hard time troubleshooting it..
1982Strand said:
okay, i tried changing line #53 as you said, but now i get another error:
Code:
/home/dan/buildtool/functions.sh: line 53: [: too many arguments
/home/dan/buildtool/functions.sh: line 64: no match: *jar
Invalid choice
#?
And with the -vx set in the shebang:
Code:
/home/dan/buildtool/functions.sh: line 53: [: too many arguments
/home/dan/buildtool/functions.sh: line 64: no match: *.jar
++ [[ -z '' ]]
++ echo 'Invalid choice'
Invalid choice
++ continue
#?
Before i changed the code, it was normal that i threw the error with the *.jars not found, but it's supposed to continue anyway..
As i said, if i quit the tool when the error came up, start the tool again and press 2, it works. (it still gives me the error with the *.jar not found, but that was normal)
I don't understand why it takes me back to the prompt as if i were about to extract the apks??
It seems to me it's because i'm still "in" the "e. extract apks from zip" - function...?? I must admit,i don't fully understand the code in that function, so i'm having a hard time troubleshooting it..
Click to expand...
Click to collapse
Sorry, I forgot using an 'if' statement in that way would produce the "too many arguments" error. This is from '*.apk' having more than one match, so this is what I came up with:
Code:
cd $IN
if [ "$(ls -1 | grep '.\+\.apk$' | wc -l)" -gt 0 ]; then #if there are more than 0 results of *.apk...
for file in *.apk ; do
echo "Decompiling $file" 2>&1 | tee -a $LOG/decompile_log.txt
apktool -q d -f $file $DEC/$file
done
cp -f $HJEM/sort.py $DEC
cd $DEC
python sort.py
rm -r sort.py
fi
if [ "$(ls -1 | grep '.\+\jar$' | wc -l)" -gt 0 ]; then #if there are more than 0 results of *jar...
for file in *jar; do
echo "Decompiling $file" 2>&1 | tee -a $LOG/decompile_log.txt
apktool -q d -f $file $DEC/$file
done
fi
After running options 'e' & 'c', then running option 2, there is no error and the script runs as it should (That is, assuming you chose option 2 of 'c' - "Clean all but apks in apk_in folder". Chosing option 1 of 'c' will obviously result in an error because no .apk or jar files will exist to decompile).
The reason you keep getting the "Invalid choice" error is because you're explicitly asking for it.
With the line #1067 mention earlier and others similar to it like the example below:
Code:
[COLOR=Gray]914[/COLOR] [[ -z $file ]] && echo "Invalid choice" && continue
When the variable string for '$file' has a zero length, as would be the case if '*jar' doesn't exist, the script will echo "Invalid choice". My example above ensures that the '$file' variable string will not have a zero length.
My suggestion would be to change "Invalid choice" to something more specific to the function for which it is being used, that way you can get a better idea of the source of your error.
As far as why you would get that error only after choosing options 'e' & 'c' and not after restarting the script, I couldn't say for sure without digging into it a little more, but at least this fixes your original problem.
I hope that helps.
Wow! That did the trick for option 2!!
But then it returns when i continue and get to option 4 "Fix sources":
Code:
[--- Fix MIUI sources ---]
...Fixing framework-miui-res.apk
patching file /home/dan/buildtool/apk_in/decompiled/framework-miui-res.apk/apktool.yml
/home/dan/buildtool/functions.sh: line 132: no match: *.rej
Invalid choice
#?
The point here is, that sometimes the patching fails and patch will generate some files (*.rej and *.orig) that needs to be deleted. But often, like most of the time really, the patching succeeds, so these files are not generated and my simple "remove" commands fail, obviously..
So, i'm guessing here that i get this zero-length issue and my script returns me to this code...
Well, this brings us back to
Code:
[[ -z $file ]] && echo "Invalid choice" && continue
From the "e" option..
This seems to be it.. I'm not entirely sure about what the code exactly means, except the "invalid choice" and continue..
The point with the option "e. Extract apks from zip", is that the user gets a list of zip files contained in the source_rom folder, then choose one.
Then a set of apks (defined in translation_list.txt) must be extracted to apk_in.
The function should simply give the options to choose a zip, x to return to the main menu, or write "invalid choice" if wrong key is entered...
But something seems broken in code, i just can't figure out what and where...
Btw, the code itself is from stackoverflow.com, so i didn't write it myself like that, i just used it for my script as it seemed to do what i needed, but i guess it needs some adjustment still
Okay, in the previous example, you had a 'for' loop that was written like this...
Code:
for file in *.apk *jar; do
this
and that
done
...which is saying, For every file (one at a time) in the current directory that matches the patterns *.apk and *jar, assign that filename to the variable '$file', then do..."this and that" while plugging in the value of '$file' for that particular iteration of the loop to the set of commands represented by "this and that". If for some reason, say, no files match the pattern *jar, then for each iteration of the loop regarding that pattern, $file will be equal to ' ' instead of something like 'filename.jar'. That is a variable string length of 0.
Code:
[[ -z $file ]] && echo "Invalid choice" && continue
...what that is, is a test to see if the string (or in this case filename) represented by '$file' has a zero length as with the example above where there were no jar files to assign to the variable '$file'. Similarly, it would most likely be the case with option 4 when there is no match for '*.rej' & '*.orig'.
In my example:
Code:
if [ "$(ls -1 | grep '.\+\.apk$' | wc -l)" -gt 0 ]; then
...I'm testing to see if the output of the command 'ls -1 | grep '.\+\.apk$' | wc -l' is greater than 0, before continuing with the 'for' loop.
The 'ls -1'command lists the contents of the current directory to one column, instead of the typical two or more. The 'wc -l' counts the number of lines in the resulting output. And grep '.\+\.apk$' is a regular expression that makes sure the resulting output only contains filenames that end in '.apk'. So if there are files that end in .apk, then the output of the command 'ls -1 | grep '.\+\.apk$' | wc -l' would be greater than 0, and the same would hold true for jar files. I'm sure there's a more elegant way of doing it, but it works.
Side note: Regular Expressions are powerful pattern matching tools that you definitely need to learn if you want to get the most our of your shell scripts. Google "regexp" or "Bash regexp" to learn more. It can be very confusing to understand at first, but once you get the hang of it, it is really pretty easy.
Anyway, getting back on track...
After running option 2 to decompile, then running option 4 to fix MIUI sources, everything runs fine...even with, or without '*.rej' & '*.orig'. I'll use the following debug as an example:
Code:
+ echo '...Fixing framework-miui-res.apk'
...Fixing framework-miui-res.apk
+ echo ''
+ patch -i /home/soup/buildtool/src_fix/framework-miui-res/apktool.diff /home/soup/buildtool/apk_in/decompiled/framework-miui-res.apk/apktool.yml
patching file /home/soup/buildtool/apk_in/decompiled/framework-miui-res.apk/apktool.yml
+ cd /home/soup/buildtool/apk_in/decompiled/framework-miui-res.apk/
[COLOR=Red]# notice there are no files that match '*.rej' or '*.orig' [/COLOR]
+ rm -f -r '*.rej'
+ rm -f -r '*.orig'
[COLOR=Red]# and there are no errors as a result of it[/COLOR]
+ echo ''
+ echo '...Fixing MiuiCompass.apk'
...Fixing MiuiCompass.apk
+ echo ''
+ patch -i /home/soup/buildtool/src_fix/MiuiCompass/apktool.diff /home/soup/buildtool/apk_in/decompiled/MiuiCompass.apk/apktool.yml
patching file /home/soup/buildtool/apk_in/decompiled/MiuiCompass.apk/apktool.yml
Hunk #1 FAILED at 4.
1 out of 1 hunk FAILED -- saving rejects to file /home/soup/buildtool/apk_in/decompiled/MiuiCompass.apk/apktool.yml.rej
[COLOR=Red]# here, there is now a match available for 'apktool.yml.rej' but not 'apktool.yml.orig'[/COLOR]
+ cd /home/soup/buildtool/apk_in/decompiled/MiuiCompass.apk/
+ rm -f -r apktool.yml.rej
+ rm -f -r apktool.yml.orig
[COLOR=Red]# still no error[/COLOR]
+ echo ''
...so I wouldn't know what to tell you without being able to recreate it on my end.
It may be helpful to give your variables $file and $zip in those functions a non-zero value after they are run to make sure there isn't any zero length hangover from a previous option, like so...
Code:
pull () {
shopt -s failglob
echo "[--- Choose rom zip to extract from, or x to exit ---]"
echo ""
echo ""
select zip in $SRC/*.zip
do
[[ $REPLY == x ]] && . $HJEM/build
[[ -z $zip ]] && echo "Invalid choice" && continue
echo
for apk in $(<$HJEM/translation_list.txt); do
unzip -j -o -q $zip system/app/$apk -d $IN 2&>1 > /dev/null;
done
unzip -j -o -q $zip system/framework/framework-res.apk -d $IN 2&>1 > /dev/null;
unzip -j -o -q $zip system/framework/framework-miui-res.apk -d $IN 2&>1 > /dev/null;
done
zip=dummy [COLOR=Red]<-- after the script is run, assign the string 'dummy' to $zip[/COLOR]
}
soupmagnet said:
In my example:
Code:
if [ "$(ls -1 | grep '.\+\.apk$' | wc -l)" -gt 0 ]; then
...I'm testing to see if the output of the command 'ls -1 | grep '.\+\.apk$' | wc -l' is greater than 0, before continuing with the 'for' loop.
The 'ls -1'command lists the contents of the current directory to one column, instead of the typical two or more. The 'wc -l' counts the number of lines in the resulting output. And grep '.\+\.apk$' is a regular expression that makes sure the resulting output only contains filenames that end in '.apk'. So if there are files that end in .apk, then the output of the command 'ls -1 | grep '.\+\.apk$' | wc -l' would be greater than 0, and the same would hold true for jar files. I'm sure there's a more elegant way of doing it, but it works.
Side note: Regular Expressions are powerful pattern matching tools that you definitely need to learn if you want to get the most our of your shell scripts. Google "regexp" or "Bash regexp" to learn more. It can be very confusing to understand at first, but once you get the hang of it, it is really pretty easy.
Click to expand...
Click to collapse
Really good stuff! Learning a lot from this, thanks! I'll dig into regular expressions right away
Anyways, i think i got around the missing *.rej and *.orig by approaching the operation with SED instead of PATCH..
First of all, it makes my code shorter and i don't need an external .diff file for the operation to succeed. (Given that i write the SED code correctly of course..)
It may be helpful to give your variables $file and $zip in those functions a non-zero value after they are run to make sure there isn't any zero length hangover from a previous option, like so...
Code:
pull () {
shopt -s failglob
echo "[--- Choose rom zip to extract from, or x to exit ---]"
echo ""
echo ""
select zip in $SRC/*.zip
do
[[ $REPLY == x ]] && . $HJEM/build
[[ -z $zip ]] && echo "Invalid choice" && continue
echo
for apk in $(<$HJEM/translation_list.txt); do
unzip -j -o -q $zip system/app/$apk -d $IN 2&>1 > /dev/null;
done
unzip -j -o -q $zip system/framework/framework-res.apk -d $IN 2&>1 > /dev/null;
unzip -j -o -q $zip system/framework/framework-miui-res.apk -d $IN 2&>1 > /dev/null;
done
zip=dummy [COLOR=Red]<-- after the script is run, assign the string 'dummy' to $zip[/COLOR]
}
Click to expand...
Click to collapse
Added the code
For now, the script runs through all options fine without halting. Great! But i still need to test it more thoroughly.
Now, i will look into refining the script. Especially the "5. mods" and "10. build flashable zip".
Here's how i'd like it to operate:
When option 4 is processed, i'd like to be able to add some modding to the files. I think it's better to do this before recompiling (option 6/12) because if an apk needs to be edited, it's already decompiled.
For the 3way reboot, it needs to modify some jars. I'd like the user to choose which zip from the source_roms folder to work with and extract the version number the zip filename. (The filename will ALWAYS contain a version number..) So that when the jars are processed, the output files will be placed in a folder with the version number (like /out/"version") This is because, when i want to build my flashable zip, i want the user to input which version number to build it for and then it would pull whatever mods are made for this version number. (Because the entire ROM would probably break if those version numbers don't match)
For the crt-off effect, it should do the same, but it has to check wether a jar file is already existing in /out/"version" and modify that one if it is. (Both mods need to modify the same file)
Right now, i have extra options for OFFICIAL roms. The mods are exactly the same, only the file naming in the function are different. I'd like to eliminate those options, by having the user choose what file to process, like i explain in the above..
Guess that gets a little complicated, hope you get what i mean.. It'll take some time to re-write my functions and the code, but eventually, i'll get there!
Ok, so I got it working, with the creation of the folder from the filename. Cool, one step further, I'll continue development tonight or tomorrow
Ok, next problem
In the following function, i want the script to check, if any apks exist in the folder. If yes, present the menu to choose which one to decompile. If no, display an error message and return to the main menu.
But something is up with the LS command, no matter if there are files or not in the folder, it returns the message "no files found"..
Having trouble figuring this one out..
Code:
decompile_single () {
shopt -s failglob
echo "[--- Choose apk number, or x to exit ---]"
echo ""
echo ""
cd $IN
if [ "$(ls -A $IN)" ]; then
echo ""
echo "No files found.."
echo ""
else
select file in *.apk
do
cat /dev/null > $LOG/decompile_log.txt
[[ $REPLY == x ]] && . $HJEM/build
[[ -z $file ]] && echo "Invalid choice for single decompiling" && continue
echo
echo "Decompiling $file" 2>&1 | tee -a $LOG/decompile_log.txt
apktool d -f "$file" $DEC/$file
cp -f $HJEM/sort.py $DEC/$file
python $DEC/$file/sort.py
rm -r $DEC/$file/sort.py
break
done
fi
}
Any ideas?
Okay...
The output of the command "ls -A $IN" exits with 0 if successful, otherwise it exits with 1. A good way to look at 'if' constructs is...(if "0" (goto)-> then.....if "anything else" (goto)-> else). If you're unsure of what the exit status of a command is, you can enter it in the terminal while piping it into the "echo $?" command. "$?" is a bash variable that represents the exit code of the previous command only.
For your example, you can test the exit status of that command like so...
Code:
ls -A ~/buildtool/apk_in | echo $?
Since the exit status is 0, then your output will be "No files found.." But here's where it gets tricky...The exit status of the 'ls' command will always be 0 unless the directory just cannot be accessed, which is why you will always get the same output..."No files found..". You can find out more about the exit status of a command by visiting its man page (man ls).
To get around this, you need to write the command in such a way that will give you an exit status of anything other than 0 if the condition is not met. Since you only want to check for the existence of ".apk" files you could expand on the command using a regular expression and the 'wc' command, like with my previous example...
Code:
if [ "$(ls -A $IN | grep '.\+\.apk$' | wc -l)" -eq 0 ]; then
echo ""
echo 'No ".apk" files found..'
echo ""
else
...
soupmagnet said:
Okay...
The output of the command "ls -A $IN" exits with 0 if successful, otherwise it exits with 1. A good way to look at 'if' constructs is...(if "0" (goto)-> then.....if "anything else" (goto)-> else). If you're unsure of what the exit status of a command is, you can enter it in the terminal while piping it into the "echo $?" command. "$?" is a bash variable that represents the exit code of the previous command only.
For your example, you can test the exit status of that command like so...
Code:
ls -A ~/buildtool/apk_in | echo $?
Since the exit status is 0, then your output will be "No files found.." But here's where it gets tricky...The exit status of the 'ls' command will always be 0 unless the directory just cannot be accessed, which is why you will always get the same output..."No files found..". You can find out more about the exit status of a command by visiting its man page (man ls).
To get around this, you need to write the command in such a way that will give you an exit status of anything other than 0 if the condition is not met. Since you only want to check for the existence of ".apk" files you could expand on the command using a regular expression and the 'wc' command, like with my previous example...
Code:
if [ "$(ls -A $IN | grep '.\+\.apk$' | wc -l)" -eq 0 ]; then
echo ""
echo 'No ".apk" files found..'
echo ""
else
...
Click to expand...
Click to collapse
Yes, thankyou!! Again, learning new stuff..
I just added your earlier code to this function.. Of course, it works like a charm! Facepalm on me! Hehe!
Been reading page after page about regular erxpressions, tests and all kinds of commands this weekend, i kinda stares blind at my code sometimes, haha!
1982Strand said:
Yes, thankyou!! Again, learning new stuff..
I just added your earlier code to this function.. Of course, it works like a charm! Facepalm on me! Hehe!
Been reading page after page about regular erxpressions, tests and all kinds of commands this weekend, i kinda stares blind at my code sometimes, haha!
Click to expand...
Click to collapse
I would say, the things you need to be comfortable with are (in order of importance IMO)...
man pages
exit statuses
debugging
regular expressions
pipes
data manipulation
loops
conditions
everything else
soupmagnet said:
I would say, the things you need to be comfortable with are (in order of importance IMO)...
man pages
exit statuses
debugging
regular expressions
pipes
data manipulation
loops
conditions
everything else
Click to expand...
Click to collapse
Got some more reading ahead of me
Anyways, i think the script is pretty good now, it suits my needs so far and most of the errors are taken care of..
I can always improve the code, so i'll probably continue developing on this.. Also because it's not perfect at all and still got some flaws here and there...

Guide: Fix WhatsApp Images mess after restoring backup

Hey there
Thought this might help some of you finding yourself in a huge mess because WhatApp restores all its media with the date the restore takes place. I wrote a little shellscript you can run in "adb shell". Just paste it into the shell and run it inside WhatsApps Folders.
In "WhatsApp Images" and it's subfolder "Sent" use this:
Code:
for file in *; do
if [ -f "$file" ]; then
timestamp=$(echo "$file" | sed 's/^IMG-\(.*\)-WA\(...\)\(.\)\..*/\10\2\.0\3/g')
echo $timestamp
echo $file
touch -m -t $timestamp $file
fi
done
For "WhatsApp Animated Gifs" and "WhatsApp Video" use this:
Code:
for file in *; do
if [ -f "$file" ]; then
timestamp=$(echo "$file" | sed 's/^VID-\(.*\)-WA\(...\)\(.\)\..*/\10\2\.0\3/g')
echo $timestamp
echo $file
touch -m -t $timestamp $file
fi
done
Keep the following in mind:
- This code works only if your version of Android allows you to change the modified date of files without root. As far as I know this is the case since Andoid 9
- This code doesn't work if you have files with a WA-Number higher than 0599 as my script doesn't take care to match the number with a proper timeformat.
- This code changes the last modified date of your files like this:
Filename: IMG-20190915-WA0039.jpg
Timestamp: 201909150003.09 -> 20190915 00:03:09
Feedbacks and comments welcome
Greets Air

A script to get a random file

Dear developers,
I have an issue in my script switching from Android 9 to 10 (devices from a Umidigi s3 Pro to a Umidigi F2)
I have installed Bosybox App on the first and Busybox Magisk module on the latter
Now the script does not work because the command
list=(`busybox find "$dirs" -type f -name *.$ext`)
returns an empty array
This is the complete script:
Bash:
#!/system/bin/sh
echo
if test "$1" = ""; then
echo "Randomfile script by Uranya <[email protected]> v1.4 01.01.2021"
echo "Usage:"
echo "sh randomfile.sh <sourcedir> <extension> <destdir>"
exit 1
fi
dirs=$1
ext=$2
dird=$3'/'
dest=$dird'random'
delim1=""
delim2=""
last='last.txt'
# create filename's array
IFS=$'\n'
# here we have the ISSUE
list=(`busybox find "$dirs" -type f -name *.$ext`)
# count number of files
num=${#list[@]}
# initialize random generator
RANDOM=$$
# generate random number in range 1-NUM
let "ran=(${RANDOM} % ${num})+ 1"
echo Random from $num files is $ran
sour=${list[ran]}
sourn=${sour#$dirs}
sourn=${sourn:1:${#sourn}}
date=$(date +"%Y.%m.%d %H:%M")
day=$(date +"%d")
hour=$(date +"%H")
minute=$(date +"%M")
message='---------------------------------------\n'$date' - '$num' >>> '$ran'\n'$delim1$sourn$delim2
if ([ "$day" = "01" ] && [[ "$minute" < "29" ]]) || [ ! -f $dird$last ]; then
echo >$dird$last $message
else
sed -i '1i'$message $dird$last
fi
echo $delim1$sourn$delim2
# rename the old file
cp $dest.$ext $dest'_back.'$ext
# copy the file
cat "$sour" >$dest.$ext
echo File copied as $delim1$dest.$ext$delim2
Can you please help me why this happens, and how to fix it?
Thank you very much for your attention!
Uranya said:
[...]
Click to expand...
Click to collapse
Having done some tests I have found this:
opening a root privileged terminal and executing
---
echo `find /storage/7BC3-1805/Music/MP3/Abba -type f -name *.mp3`
---
it returns two strings containing the names of files inside that folder, but putting it in my script continues to return an empty array, so the issue is not in the access to the folder, but in the syntax, I guess
try putting that *.$ext into quotes...
Dear friends, CXZa, after a couple of hours debugging the script, finally, I have found the mistake!
The line to be used is:
list=( `find "$dirs" -type f -name "*.$ext"` )
it is a very subtle difference: the space after and before the parenthesis!
(even the word busybox is useless)
Oddly in the Busybox app (I have had on my S3 Pro) the spaces are not mandatory, whilst in the Busybox Magisk module those spaces ARE mandatory!
I'm using that script for almost 8 years to have an every day different music for my wake up.
I'm using Tasker to call it just before my alarm get off, so the same file contains every day, a different song.
I have done a change also in the array index that did not began by 0...
So, here it is the right script:
Bash:
#!/system/bin/sh
echo
if test "$1" = ""; then
echo "Randomfile script by Uranya <@uranya7x> v1.5 26.03.2021"
echo "Usage:"
echo "sh randomfile.sh <sourcedir> <extension> <destdir>"
exit 1
fi
dirs=$1
ext=$2
dird=$3'/'
dest=$dird'random'
delim1=""
delim2=""
last='last.txt'
# create filename's array
IFS=$'\n'
list=( `find "$dirs" -type f -name "*.$ext"` )
# count number of files
num=${#list[@]}
# generate random number in range 1-NUM
let "ran=(${RANDOM} % ${num})+ 1"
echo Random from $num files is $ran
sour=${list[$ran-1]}
sourn=${sour#$dirs}
sourn=${sourn:1:${#sourn}}
date=$(date +"%Y.%m.%d %H:%M")
day=$(date +"%d")
hour=$(date +"%H")
minute=$(date +"%M")
message='---------------------------------------\n'$date' - '$num' >>> '$ran'\n'$delim1$sourn$delim2
if ([ "$day" = "01" ] && [[ "$minute" < "29" ]]) || [ ! -f $dird$last ]; then
echo >$dird$last $message
else
sed -i '1i'$message $dird$last
fi
echo $delim1$sourn$delim2
# rename the old file
cp $dest.$ext $dest'_back.'$ext
# copy the file
cat "$sour" >$dest.$ext
echo File copied as $delim1$dest.$ext
I hope it will be useful to someone else that loves to be waked up by music...
Peace everywhere!

Categories

Resources