I built a system that simulates memory paging, just like an MMU.
And to better regulate and understand how it works, I am logging it.
My problem is that it seems the log is not accurately reflecting the operations of the system, or rather it does, but then I have a big problem with threads that I need some help solving.
I'll try and explain.
public void run() //gets pages and writes to them
{ // i printed the pageId of every process to check they are running at the same time and competing for resources
for(ProcessCycle currentCycle : processCycles.getProcessCycles())
{
Long[] longArray = new Long[currentCycle.getPages().size()];
try {
for(int i = 0; i < currentCycle.getPages().size();i++)
{
MMULogger.getInstance().write("GP:P" + id + " " + currentCycle.getPages().get(i) + " " + Arrays.toString(currentCycle.getData().get(i)), Level.INFO);
}
Page<byte[]>[] newPages = mmu.getPages(currentCycle.getPages().toArray(longArray));
List<byte[]> currentPageData = currentCycle.getData();
System.out.println("process id " + id);
for(int i = 0; i < newPages.length;i++)
{
byte[] currentData = currentPageData.get(i);
newPages[i].setContent(currentData);
}
Thread.sleep(currentCycle.getSleepMs());
} catch (ClassNotFoundException | IOException | InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
this code snippet is taken from a class called Process. Just like in a computer, I have multiple processes, and they each need to read and write from pages, which they request from the class called MMU. this is the "mmu.getpages" part.
We also write to our log file inside the method get pages:
public synchronized Page<byte[]>[] getPages(java.lang.Long[] pageIds) throws java.io.IOException, ClassNotFoundException
{
#SuppressWarnings("unchecked")
Page<byte[]>[] toReturn = new Page[pageIds.length];
for(int i = 0; i < pageIds.length; i++)
{
Long currentPage = algo.getElement(pageIds[i]);
if(currentPage == null) //page not found in RAM
{
if(ram.getInitialCapacity() != ram.getNumOfPages()) //ram is NOT full
{
MMULogger.getInstance().write("PF:"+pageIds[i], Level.INFO);
algo.putElement(Long.valueOf(pageIds[i]),Long.valueOf(pageIds[i]));
ram.addPage(HardDisk.getInstance().pageFault(pageIds[i]));
}
else //ram is full
{
Long IDOfMoveToHdPage = algo.putElement(pageIds[i], pageIds[i]);
Page<byte[]> moveToHdPage = ram.getPage((int)((long)IDOfMoveToHdPage));
Page<byte[]> moveToRAM = HardDisk.getInstance().pageReplacement(moveToHdPage, pageIds[i]);
ram.removePage(moveToHdPage);
ram.addPage(moveToRAM);
MMULogger.getInstance().write("PR: MTH " + moveToHdPage.getPageId() + " MTR " + moveToRAM.getPageId(), Level.INFO);
}
}
toReturn[i] = ram.getPage((int)((long)pageIds[i]));
}
return toReturn;
}
So all in all to recap - a process requests pages, I write to the log file which process requests which page and what it wants to write to it, and then I call mmu.getpages, and the logic of the system continues.
My problem is that the log looks like this:
GP:P2 5 [102, 87, -9, 85, -5]
GP:P1 1 [-9, -18, 50, -124, -102]
GP:P4 10 [79, -51, 67, 118, 111]
GP:P2 6 [-20, -22, 3, -74, -65]
GP:P3 7 [90, 56, 91, 71, -115]
PF:5
GP:P6 18 [28, -39, -3, 64, -117]
GP:P5 13 [72, -26, 52, -84, 6]
GP:P4 11 [-55, -70, -88, -9, 38]
GP:P1 2 [39, 112, -117, 5, 109]
GP:P5 12 [38, -31, 18, -40, 36]
which is not what I wanted. At first you can see process 2 requested page 5 and wanted to write to it [102, 87, -9, 85, -5].
After that line, I would have expected to see "PF:5" but its further down. I think it is the case because process 2 ran out of time and didnt manage to finish mmu.getpages operation. so it never printed PF:5 to the file.
That is a problem for me. I want the processes to run simultaneously in a multithreaded fashion, but i want the log to be of the form:
GP:P2 5 [1,1,1,1,1]
PF:5
GP:P2 7 [1,2,3,4,5]
PF:7
GP:P19 12 [0,0,0,0,0]
PF:12
For example
Related
I'm writing application to mesure speed of CRUD with hibernate for Derby.
This is my function :
#Override
public BulkTestResult testBulkInsertScenario(Long volume, Integer repeat) {
StopWatch sw = new StopWatch();
BulkTestResult bulkTestResult = new BulkTestResult();
bulkTestResult.setStartDate(Instant.now());
bulkTestResult.setCountTest(volume);
bulkTestResult.setTestRepeat(repeat);
familyService.clear();
for(int i =0; i < repeat; i++) {
List<ProjectEntity> projects = dataAnonymization.generateProjectEntityList(volume);
runBulkTest(sw, bulkTestResult, projects, true);
}
bulkTestResult.setEndDate(Instant.now());
return bulkTestResult;
}
private void runBulkTest(StopWatch sw, BulkTestResult bulkTestResult, List<ProjectEntity> projects, boolean resetAfter) {
sw.reset();
sw.start();
familyService.save(projects);
sw.stop();
bulkTestResult.addMsSpeedResult(sw.getTime());
if (resetAfter) familyService.clear();
sw.reset();
}
clear method remove all record from DB.
The problem that I have is values that I recieved as output of application.
Testing data : 1000 record, and 10 repeats
Example speed values recieved running this test few times:
311, 116, 87, (...)38
32, 27, 30, (...) 24
22, 19, 18, (...) 21
19, 18, 18, (...) 19
Why there are so many difference and why for first time insert is always slower ?
It could be any hardware acceleration ?
I found solution.
This issue is related to optimalization. After Disable JIT, recieved values are correct.
-Djava.compiler=NONE -Xint
I'm trying to store a symmetric key (SecretKey which is Triple-DES key, ECB mode) on a cryptographic USB token. I use the following code to do it:
private void storeSecretKey( SecretKey secretKey, String alias ) throws StoreException {
try {
log.info("Format: " + secretKey.getFormat());
log.info("Alg: " + secretKey.getAlgorithm());
log.info("STORE KEY (bytes): " + Arrays.toString( secretKey.getEncoded()));
log.info("STORE KEY: " + ArrayUtils.convertToHexString( secretKey.getEncoded(), false, false));
myKeyStore.setKeyEntry(alias, secretKey, tokenPIN.toCharArray(), null);
myKeyStore.store(null);
Key key = myKeyStore.getKey(alias, tokenPIN.toCharArray());
log.info("Format: " + key.getFormat());
log.info("Alg: " + key.getAlgorithm());
log.info("FINAL KEY (bytes): " + Arrays.toString( key.getEncoded()));
log.info("FINAL KEY: " + ArrayUtils.convertToHexString( key.getEncoded(), false, false));
}
catch ( KeyStoreException e ) {
throw new StoreException( "Unable to store encryption key", e );
} catch (NoSuchAlgorithmException e) {
e.printStackTrace();
} catch (UnrecoverableEntryException e) {
e.printStackTrace();
}
}
And I get the following logging:
INFO: Format: RAW
INFO: Alg: DESede
INFO: STORE KEY (bytes): [87, -81, -89, -62, 5, -116, -46, 111, -85, -52, -28, -85, -26, -57, -26, -58, -66, -52, -16, 30, 89, -45, 61, -86]
INFO: STORE KEY: 57afa7c2058cd26fabcce4abe6c7e6c6beccf01e59d33daa
INFO: Format: RAW
INFO: Alg: DESede
INFO: FINAL KEY (bytes): [87, -82, -89, -62, 4, -116, -45, 110, -85, -51, -27, -85, -26, -57, -26, -57, -65, -51, -15, 31, 88, -45, 61, -85]
INFO: FINAL KEY: 57aea7c2048cd36eabcde5abe6c7e6c7bfcdf11f58d33dab
The key that is stored on the token (STORE KEY) and the key that I get using getKey() (FINAL KEY) are different. How can it be?
Please inform me if you need any other information it might be missing.
Thank you.
Your USB token simply adjusts the parity of the 3DES key. Each lower end bit of each byte is a parity bit for (triple) DES.
It needs to be set if the addition of the 7 highest bits is an even number, and unset if the number is odd. In the end each byte should have odd parity if you add all the bits together. So the first byte with value 0x57 or 0b0101011_1 and has 4 bits in the highest positions so the final bit has to be 1 - but it already is set, so no adjustment is necessary. The second byte, 0xAF or 0b1010111_1 has 5 bits in the highest position, so the last bit needs to be 0. This is not the case, so it is adjusted to 0b1010111_0 or 0xAE.
If you want to have the same value then you can construct your (triple) DES keys using a SecretKeyFactory instead of using a random number generator directly. The SecretKeyFactory will also adjust the parity for you - no need to program it yourself. I'd recommend this as other implementations could reject bytes where the number of bits is even (although generally the bits are adjusted or ignored). As indicated by James, the lowest bits are not used by (triple) DES during encryption / decryption.
This will create correctly coded triple DES keys (168 bit effective, 192 bits encoded). These are also called DES ABC keys as all three DES keys are different (with a very high probability).
public static SecretKey generate192Bit3DESKey() {
KeyGenerator keyGen;
try {
keyGen = KeyGenerator.getInstance("DESede");
} catch (NoSuchAlgorithmException e) {
throw new IllegalStateException("DESede functionality is required for Java, but it is missing");
}
// NOTE: this is the effective key size excluding parity bits
// use 112 for two key (ABA) triple DES keys (not recommended)
keyGen.init(168);
// this does adjust parity
SecretKey desABCKey = keyGen.generateKey();
return desABCKey;
}
If your data needs adjusting afterwards, you can use this (with only a for loop, no additional branching):
public static byte[] adjustDESParity(final byte[] keyData) {
for (int i = 0; i < keyData.length; i++) {
// count the bits, and XOR with 1 if even or 0 if already odd
keyData[i] ^= (Integer.bitCount(keyData[i]) % 2) ^ 1;
}
return keyData;
}
i want to get a list which contains entire OrderInfo Object, for example, if i do this, it is result which the result i want to.
def find_all(self):
result_list = []
orderDao = DaoUtil.DaoGeneric()
session = orderDao.getSession()
try:
for row in session.query(OrderInfo).all():
result_list.append({
'id':row.id,
'name': row.name,
'age': row.age,
'create_time': row.create_time.strftime("%Y-%m-%d %H:%M:%S"),
'update_time': row.update_time.strftime("%Y-%m-%d %H:%M:%S"),
'version': row.version
})
session.commit()
except Exception, e:
print e
session.rollback()
return result_list
but i want to a list which contains OrderInfo object from the query, because the result which the query return have other columns (the simple list of all DeclarativeBase's instances.) except OrderInfo{id,name,age,create_time,update_time,version}, the query do not return OrderInfo object directly. the following which i want to:
def find_all(self):
result_list = []
orderDao = DaoUtil.DaoGeneric()
session = orderDao.getSession()
try:
for row in session.query(OrderInfo).all():
result_list.append(row.orderInfo) // if the row has a property for orderInfo Object, because the result which java can achieve , the example for java is : List<OrderInfo> orderList = session.query(); please help to achieve it
session.commit()
except Exception, e:
print e
session.rollback()
return result_list
beacause i use sqlalchemy in python just now, i am not very sure. How to get a list which contains OrderInfo Object from query in sqlalchemy
#univerio
for row in session.query(OrderInfo).all():
the line of variable 's row which includes the following column:
_decl_class_registry,
_sa_class_manager,
_sa_instance_state,
metadata,
query,
id,
name,
age,
create_time,
update_time ,
version
only this (OrderInfo{id, name, age, create_time,update_time,version}) is what i want to get, other columns which i do not want to get.
OrderInfo:
from sqlalchemy import Column, Integer, String, Date, DateTime
from sqlalchemy.ext.declarative import declarative_base
Base=declarative_base()
class OrderInfo(Base):
__tablename__ = 'order_info'
# __table__ = 'order_info'
id = Column(Integer, primary_key=True)
name = Column(String(100))
age = Column(Integer)
create_time = Column(Date)
update_time = Column(Date)
version = Column(Integer)
#univerio, i write here because the comments has limit of words, it achieve this result, like this:
for row in session.query(OrderInfo).all():
result_list.append({
'id':row.id,
'name': row.name,
'age': row.age,
'create_time': row.create_time.strftime("%Y-%m-%d %H:%M:%S"),
'update_time': row.update_time.strftime("%Y-%m-%d %H:%M:%S"),
'version': row.version
})
the OrderInfo has very few columns, if the OrderInfo object has more than serveral hundred columns， but it will take a long time, so i want find the result which can simple achieve this function that is similar to List in Java
#univerio,
i find the answer which i want to get:
def find_all(self):
result_list = []
orderDao = DaoUtil.DaoGeneric()
session = orderDao.getSession()
try:
for row in session.query(OrderInfo).all():
result_list.append(DictUtil.object_as_dict(row))
session.commit()
except Exception, e:
print e
session.rollback()
return result_list
def object_as_dict(obj):
result = {instance.key: getattr(obj, instance.key) for instance in inspect(obj).mapper.column_attrs}
print result
return result
output :
[
{'updateTime': datetime.datetime(2017, 6, 15, 13, 56, 16), 'bankName': u'ICBC', 'bankNo': u'6228480666622220011', 'createTime': datetime.datetime(2017, 6, 15, 13, 56, 16), u'version': 0, u'id': 1},
{'updateTime': datetime.datetime(2017, 6, 15, 13, 57, 40), 'bankName': u'ICBC', 'bankNo': u'6228480666622220011', 'createTime': datetime.datetime(2017, 6, 15, 13, 57, 40), u'version': 0, u'id': 2},
{'updateTime': datetime.datetime(2017, 6, 15, 13, 58), 'bankName': u'ICBC', 'bankNo': u'6228480666622220011', 'createTime': datetime.datetime(2017, 6, 15, 13, 58), u'version': 0, u'id': 3}
]
i have two values:
String a = "00tz"; // (Eclipse internal debug value: [, 0, 0, t, z]) and
String b = "tz"; // (Eclipse internal debug value: [, t, z])
I am reading this values from an ArrayList like
for (String a : stringLists) {
...
}
I get "false" when i compare this two values with StringUtils.containsIgnoreCase(a,b). But it should return true because "tz" is existing in "00tz".
Im using apache.commons.lang3.StringUtils. To change the values a & b didn't worked. The length of "a" is 5 and "b" is 3. It also returns false when i use a.contains(b).
These are the results when i output the value with
System.out.println(Arrays.toString(a.getBytes(StandardCharsets.UTF_8)));
a:[-17, -69, -65, 48, 48, 118, 119]
b:[-17, -69, -65, 118, 119]
Im reading this values from a .txt file which contains several values like this. I read it in this way:
File fileA = new File("test/a.txt");
File fileB = new File("test/b.txt");
lista = (ArrayList<String>) FileUtils.readLines(fileA, "utf-8");
listb = (ArrayList<String>) FileUtils.readLines(fileB, "utf-8");
Do you have an idea what the problem is?
Thank you!
I'm creating a 24bit bmp which in general works fine (I've been using this functionality for some time). Now I've tried to write a bmp with 970 x 970 pixels and I'm ending up in corrupt files (I've exported bigger images before, I'm having problems with this particular resolution).
This is how I build the header:
private static byte[] createHeader(int width, int height) {
int size = width * height * 3 + 54;
byte[] ret = new byte[54];
set(ret, 0, (byte) 'B', (byte) 'M');
set(ret, 2, intToDWord(size));
set(ret, 6, intToDWord(0));
set(ret, 10, intToDWord(54));
set(ret, 14, intToDWord(40));
set(ret, 18, intToDWord(width));
set(ret, 22, intToDWord(height));
set(ret, 26, intToWord(1));
set(ret, 28, intToWord(24));
set(ret, 30, intToDWord(0));
set(ret, 34, intToDWord(width * height * 3));
set(ret, 38, intToDWord(0));
set(ret, 42, intToDWord(0));
set(ret, 46, intToDWord(0));
set(ret, 50, intToDWord(0));
return ret;
}
Here is the resulting image (this test image should be completely red):
test_corrupt.bmp (2.6mb)
I've analyzed the header, checked the size, I can't find the reason why this image is not a valid BMP.
Does anyone have a clue? I'm not making any progress.
It may be because of BMP files expects row lengths to be a multiple of 4 bytes. This changes the size you specified in header offset 34 and therefore the size in offset 2. Please refer to the following for details:
http://en.wikipedia.org/wiki/BMP_file_format
Related Part:
For file storage purposes, only the size of each row must be a multiple of 4 bytes while the file offset can be arbitrary
You can compare file by creating a 970x970 Red BMP file using MS Paint.