Good day!
I'm using cloud dlp api to inspect bigquery views by converting chunks of the data into ContentItem and passing it to the inspect request. However, I am having trouble converting the findings and saving it to a bigquery table. Before, I used an airflow DLP operator for this and it is being done automatically by passing output storage config in an InspectConfig. However, that approach won't be applicable anymore as I'm calling the DLP api per chunks of data using apache beam in java.
I saw that the finding object has a writeTo() method but I'm not sure how to use it and how to save the findings with correct types into a bigquery table. can you help me with this? I'm currently stuck. thank you!
what I want to do is something like this
for (Finding res : result.getFindingsList()){
TableRow bqRow = new TableRow();
Object data = res.getLocation();
bqRow.set("field", data);
context.output(bqRow);
}
but this approach wouldn't save it in bigquery with correct types, especially for getLocation as it returns something like a protobuf message type.
I was trying to see if I can use the writeTo() method but I'm not sure how to use it. Thank you in advance for the help!
for (Finding res : result.getFindingsList()){
res.writeTo(...)
...
context.output(...);
}
If you use HybridInspect we'll store the findings for you to BigQuery.
https://cloud.google.com/dlp/docs/how-to-hybrid-jobs
If you do it yourself you will need to convert to a native BQ format like json
Load protobuf data to bigquery
I have developed and released a game using LibGDX platform for Android. I'm using Json to save and load information:
To save an object (a rocket class):
Json json = new Json();
String rocketStr = json.toJson(rocket);
To load this object:
Rocket rocket = json.fromJson(Rocket.class, rocketStr);
After the release, I have added a new advertising provider (adMob) and released the update to the game. I have not made any changes to the classes which are being saved. And I have not changed any library versions. The only change was some new classes added and used in the project.
With this new update the game could no longer load the saved objects with Json - it would come up with an exception when trying to run json.fromJson(Rocket.class, rocketStr); Exceptions would differ but mainly there were two: 1) property not found or 2) unable to cast value to some property.
After some experimenting, I found out that in production Android (or LibGDX?) obscures all the class names and property names and when Json is writing a class the resulting string looks like this:
{A:97.50652,A2:{E:RIGHTTHRUSTER,F:true,a:Liberator Thrusters,b:2,d:false,f:true,g:{}},H:24.934832,I:10,J:true,K:30,M2:1587,N2:1647,O2:1527,Y:-7.3379517,a:{seed0:28965116732361103,seed1:-4728429348234121846},a0:-19.308016,b:Liberator,b0:17.0175,c:1,d:1,e:false,j0:{a:500},m:true,o:497.43192,p:1587,s:100,t:100,u2:{E:TOP,F:true,a:Liberator Nose,b:1,c:1,d:false,f:true,f0:[],g:{}},z2:{E:LEFTTHRUSTER,F:true,a:Liberator Thrusters,b:2,c:1,d:false,f:true,f0:[],g:{}}}
And after the update, the same object saved has some different field names:
{B2:{E:LEFTTHRUSTER,F:true,a:Liberator Thrusters,b:43,c:1,f:true,f0:[],g:{}},C:100,D2:{E:RIGHTTHRUSTER,F:true,a:Liberator Thrusters,b:43,f:true,g:{}},J:0.128269,K:-7.5,L:true,N:true,O2:1587,P2:1647,Q2:1527,R:-5,a0:-2.9551423,c:{seed0:-3537680883966184457,seed1:6879650071827728698},d:Liberator,e:17,f:1,l0:{a:500},o:true,q:595.54803,r:1587,u:100,v:100,w2:{E:TOP,F:true,a:Liberator Nose,b:42,c:1,f:true,f0:[],g:{}}}
For example, in first example one of the properties is called 'A2' and in the second example the same property is called 'D2'
Is my understanding correct that Android (or LibGDX?) can change the obfuscated class names for the same classes when I make some changes to the project?
And most importantly, what can I do to make sure that the class / property names stay the same to be able to save and load the same object with Json after any updates? Are there any alternatives / better methods to save / load information?
In my Android application I've created multiple .db files with each containing multiple tables.
Is there a way to copy all tables to one central file? Is this possible without having to write a for loop for each individual row, table and file? Aka is there a simple way of doing this?
As the data is stored on-device it's not possible to use any server-side mechanisms.
You can use ATTACH DATABASE instruction in SQLite. Here is documentation
When you attach the database, use something like this INSERT INTO DestinationTable SELECT * FROM attachedDatabase.SourceTable
fun migration(context: Context) {
val destinationDb = DatabaseHelper2(context)
val sourceDbPath = context.getDatabasePath("database1.sqlite")
val writableDb = destinationDb.writableDatabase
writableDb.execSQL("ATTACH DATABASE '${sourceDbPath.absolutePath}' AS attached")
writableDb.execSQL("INSERT INTO DestinationTable SELECT * FROM attached.SourceTable")
}
I am developing a GWT application in which I need to keep some state of application using Itemscript JSON library's JsonSystem.
As for the purpose I created a JsonSystem and put data inside it as follows:
GwtSystem.SYSTEM.put(GwtHttpConnector.CONFIG_URL, false);
JsonSystem system = GwtSystem.SYSTEM;
system.put("state","state_value");
I want to know,
- how long is this data will be available?
- when ever I refresh my page these data will not be available, so I doubt is this the procedure to store data in-memory, if so what is the actual procedure?
references : https://code.google.com/p/itemscript/ [3rd feature]
: https://code.google.com/p/itemscript/wiki/GettingStartedGwtJava
Apparently it doesn't say on the project page but if it is a real in-memory database then you will lose the data once you refresh the page. Did you try it ?
If you want to keep the data between refreshes you can use LocalStorage. You have to dump the JsonSystem convert to a String and store in LocalStorage.
in my application i need to copy a schema with its tables and store procedures from a base schemn to a new schema.
i am looking for a way to implement this.
i looked into exacting the mysqldump using cmd however it is not a good solution because i have a client side application and this requires an instillation of the server on the client side.
the other option is my own implantation using show query.
the problem here is that i need t implement it all from scratch and the must problematic part is that i will need to arrange the order of the tables according to there foreign key (because if there is a foreign key in the table, the table i am pointing to needs to be created first).
i also thought of creating a store procedure to do this but store procedures in my SQL cant access the disk.
perhaps someone has an idea on how this can be implemented in another way?
You can try using the Apache ddlutils. There is a way to export the ddls from a database to an xml file and re-import it back.
The api usage page has examples on how to export schema to an xml file, read from xml file and apply it to a new database. I have reproduced those functions below along with a small snippet on how to use it to accomplish what you are asking for. You can use this as starting point and optimize it further.
DataSource sourceDb;
DataSource targetDb;
writeDatabaseToXML(readDatabase(sourceDb), "database-dump.xml");
changeDatabase(targetDb,readDatabaseFromXML("database-dump.xml"));
public Database readDatabase(DataSource dataSource)
{
Platform platform = PlatformFactory.createNewPlatformInstance(dataSource);
return platform.readModelFromDatabase("model");
}
public void writeDatabaseToXML(Database db, String fileName)
{
new DatabaseIO().write(db, fileName);
}
public Database readDatabaseFromXML(String fileName)
{
return new DatabaseIO().read(fileName);
}
public void changeDatabase(DataSource dataSource,
Database targetModel)
{
Platform platform = PlatformFactory.createNewPlatformInstance(dataSource);
platform.createTables(targetModel, true, false);
}
You can use information_schema to fetch the foreign key information and build a dependency tree. Here is an example.
But I think you are trying to solve something that has been solved many times before. I'm not familiar with Java, but there are ORM tools (for Python at least) that can inspect your current database and create a complementing model in Java (or Python). Then you can deploy that model into another database.