Screen scrapping

What is Screen Scraping?

Screen Scraping means reading the contents of a web page. Suppose you go to yahoo.com, what you see is the interface which includes buttons, links, images etc. What we don't see is the target url of the links, the name of the images, the method used by the button which can be POST or GET. In other words we don't see the HTML behind the pages. Screen Scraping pulls the HTML of the web page. This HTML includes every HTML tag that is used to make up the page.

Why use screen scraping?

The question that comes to our mind is why do we ever want the HTML of any web page. Screen Scraping does not stop only on pulling out the HTML but displaying it also. In other words you can pull out the HTML from any web page and display that web page on your page. It can be used as frames. But the good thing about screen scraping is that it is supported by all browsers and frames unfortunately are not.

Also sometimes you go to a website which has many links which says image1, image2, image3 and so on. In order to see those images you have to click on the image and it will enlarge in the parent or the new window. By using screen scraping you can pull all the images from a particular web page and display them on your own page.



code
===================================================================
//Html part
//start html

----------------------------------------------------
< %@ Page language="c#" Codebehind="screenscrapping.aspx.cs" AutoEventWireup="false" Inherits="oops.screenscrapping" %>
< !DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" >
< HTML>
< HEAD>
< title> screenscrapping< /title>
< meta content="Microsoft Visual Studio 7.0" name=GENERATOR>
< meta content=C# name=CODE_LANGUAGE>
< meta content=JavaScript name=vs_defaultClientScript>
< meta content=http://schemas.microsoft.com/intellisense/ie5 name=vs_targetSchema>
< /HEAD>
< body MS_POSITIONING="GridLayout">
< form id=screenscrapping method=post runat="server">
< h1> Screen Scrape of www.yahoo.com< /H1>
< p> < asp:label id=lblHTMLOutput runat="server"> < /asp:label>
< /FORM>
< /body>
< /HTML>

//end of html----------------------------------------------------
//code-behind part
//start code behind--------------------------------------------
using System;
using System.Collections;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Web;
using System.Web.SessionState;
using System.Web.UI;
using System.Web.UI.WebControls;
using System.Web.UI.HtmlControls;
using System.Net;
using System.IO;
using System.Text;



protected System.Web.UI.WebControls.Label lblHTMLOutput;//put a web control label //with this name

private void Page_Load(object sender, System.EventArgs e)
{
// Put user code to initialize the page here

WebClient obj = new WebClient();
const string strUrl="http://yahoo.com";
byte[] reqHTML;
reqHTML = obj.DownloadData(strUrl);
UTF8Encoding objUTF8 = new UTF8Encoding();
lblHTMLOutput.Text=objUTF8.GetString(reqHTML);
}
//end of code-behind-------------------------------------------------------------------
//end of code==================================================================

ASP.Net Optimization

Design Considerations Consider security and performance. Partition your application logically. ...	Threading Tune the thread pool by using the formula to reduce contention. Consider minIoThreads and minWorkerThreads for burst load. ...
Server Controls Identify the use of view state in your server controls. Use server controls where appropriate. Avoid creating deep hierarchies of controls.	Data Binding Avoid using Page.DataBind. Minimize calls to DataBinder.Eval.
Application State Use static properties instead of the Application object to store application state. Use application state to share static, read-only data. Do not store single-threaded apartment (STA) COM objects in application state.	Session State Prefer basic types to reduce serialization costs. Disable session state if you do not use it. Avoid storing STA COM objects in session state. Use the ReadOnly attribute when you can.
String Management Use Response.Write for formatting output. Use StringBuilder for temporary buffers. Use HtmlTextWriter when building custom controls.	Exception Management Implement a Global.asax error handler. Monitor application exceptions. Set timeouts aggressively. ...
Resource Management Do not cache or block on pooled resources. Pool resources. ...	Pages Trim your page size. Enable buffering. ...
Caching Separate dynamic data from static data in your pages. Cache the right data. Use kernel caching on Microsoft® Windows Server™ 2003. ...	State Management Store simple state on the client where possible. Consider serialization costs.
View State Disable view state if you do not need it. Minimize the number of objects you store in view state. Determine the size of your view state.	HTTP Modules Avoid long-running and blocking calls in pipeline code. Consider asynchronous events.
COM Interop Use ASPCOMPAT to call STA COM objects. Avoid storing COM objects in session state or application state. Avoid storing STA components in session state. ...	Data Access Use paging for large result sets. Use a DataReader for fast and efficient data binding. Prevent users from requesting too much data. Consider caching data. ...
Security Considerations Constrain unwanted Web server traffic. Turn off authentication for anonymous access. Validate user input on the client. Avoid per-request impersonation. Avoid caching sensitive data. Segregate secure and non-secure content. Only use Secure Sockets Layer (SSL) for pages that require it. Use absolute URLs for navigation. Consider using SSL hardware to offload SSL processing. Tune SSL timeout to avoid SSL session expiration.
Deployment Considerations Avoid unnecessary process hops. Understand the performance implications of a remote middle tier. Short-circuit the HTTP pipeline. Configure the memory limit. Disable tracing and debugging. Ensure content updates do not cause additional assemblies to be loaded. Avoid XCOPY under heavy load. Consider precompiling pages. Consider Web garden configuration. Consider using HTTP compression. Consider using perimeter caching.

.Net Memory Optimization

Forcing Garbage Collection in .NET

There might be times in your application when you want to force the .NET Garbage Collector (GC) to spin through all unused objects and de-allocate them. The method for accomplishing this task is the GC.Collect method. When you call GC.Collect, the GC will run each object's finalizer on a separate thread. Therefore, another method to keep in mind is GC.WaitForPendingFinalizers. This synchronous method that will not return until the GC.Collect has finished its work.

Here's a simple example of using these two methods:

using System;
namespace GCCollect
{

  class Account
  {
    public Account(string accountNumber)
    {
     this.accountNumber = accountNumber;
      Console.WriteLine("Account::Acount - c'tor");
    }

    ~Account()
    {
      Console.WriteLine("Account::~Acount - d'tor");
    }

    protected string accountNumber;

    override public string ToString() { return accountNumber; }

  };

  class Class1
  {

    [STAThread]
    static void Main(string[] args)
    {

      CreateAccount("111006116");

      GC.Collect();

      GC.WaitForPendingFinalizers();

      Console.WriteLine("Application ending");

    }

    public static void CreateAccount(string accountNumber)
    {

      Console.WriteLine("CreateAccount - instantiate Account object");

      Account account = new Account(accountNumber);

      Console.WriteLine("CreateAccount - created account number {0}",

                        account);

    }

  }

}

Type Sizing

Consider the following C# structure (for simplicity, I have avoided specifying any access modifier for these members):

struct BadValueType
{

char c1;

int i;

char c2;
}

As with the default packing in unmanaged C++, integers are laid out on four-byte boundaries, so while the first character uses two bytes (a char in managed code is a Unicode character, thus occupying two bytes), the integer moves up to the next 4-byte boundary, and the second character uses the subsequent 2 bytes. The resulting structure is 12 bytes when measured with Marshal.SizeOf (it's also 12 bytes when measured with sizeof on the .NET Framework 2.0 running on my 32-bit machine). If I reorganize this as follows, the alignment works in my favor, resulting in an 8-byte structure:

struct GoodValueType
{

int i;

char c1;

char c2;

}

Another noteworthy point is that smaller types use less memory. That may seem obvious, but many projects use standard integers or decimal values even when they are unnecessary. In my GoodValueType example, assuming the integer values will never be greater than 32767 or less than -32768, I can cut the size of this type even further by using a short integer, as shown in the following:

struct GoodValueType2

{

    short i;

    char c1;

    char c2;

}

Properly aligning and sizing this type reduced it from 12 to 6 bytes. (Marshal.SizeOf will report 4 bytes for GoodValueType2, but that's because the default marshaling for a char is as a 1 byte value.) You will be surprised how much the size of your structures and classes can be reduced if you pay attention.

ENum

enum AddressType
{

Home,

Secondary,

Office

}

class Address
{

public bool IsPayTo;

public AddressType

AddressType;

public string Address1;

public string Address2;

public string City;

public string State;

public string Zip;

}

---

Figure 3 Reducing Type Size

enum AddressType : byte

{

Home,

Secondary,

Office

}

class Address

{

byte _isPayTo;

AddressType _addrType;

string _address1;

string _address2;

string _city;

string _state;

string _zip;

}

Singletons

public class Singleton

{

    private static Singleton _instance = new Singleton();

    public static Singleton GetInstance()

    {

        return _instance;

    }

}

The singleton pattern ensures that only a single instance of a class is normally used by an application, but still allows alternate instances to be created as required. This saves memory because the application can use the one shared instance, rather than having different components allocate their own private instances. Use of the static constructor ensures that memory for the shared instance is not allocated until some portion of the application requires it. This might be important in large applications that support many different types of functionality, since the memory for the object is only allocated if the class is actually used.

Use String.Builder

The String class provides a number of opportunities to consume large amounts of memory unintentionally. The simplest example is the concatenation of strings. Concatenating four strings incrementally (adding one string at a time to the new string) will internally produce seven string objects, since each addition produces a new string. The StringBuilder class in the System.Text namespace joins strings together without allocating a new string instance each time; this efficiency greatly improves memory utilization. The C# compiler also helps in this regard because it transforms a series of string concatenations in the same code statement into a single call to String.Concat.

String Replace

The String.Replace method provides another example. Consider a system that reads and processes a number of input files sent from an external source. These files might require preprocessing to put them into an appropriate format. For discussion purposes, suppose I had a system that had to replace each occurrence of the word "nation" with "country", and each occurrence of the word "liberty" with "freedom". This can be done quite easily with the following code snippet:

//Bad Logic

using(StreamReader sr = new StreamReader(inPath))

{

    string contents = sr.ReadToEnd();

    string result = contents.Replace("nation", "country");

    result = result.Replace("liberty", "freedom");

    using(StreamWriter sw = new StreamWriter(outPath))

    {

        sw.Write(result)

    }

}

  This works perfectly, at the expense of creating three strings that are the length of the file. The Gettysburg Address is roughly 2400 bytes of Unicode text. The U.S. Constitution is over 50,000 bytes of Unicode text. You see where this is going.

Now suppose each file was roughly 1MB of string data and I had to process up to 10 files concurrently. Reading and processing these 10 files will consume, in our simple example, around 10MB of string data. This is a rather large amount of memory for the garbage collector to allocate and clean up on an ongoing basis.

//Good Logic

static void ProcessFile(FileStream fin, FileStream fout)

{

    int next;

    while ((next = fin.ReadByte()) != -1)

    {

        byte b = (byte)next;

        if (b == 'n' || b == 'N')

          CheckForWordAndWrite(fin, fout, "nation", "country", b);

        else if (b == 'l' || b == 'L')

          CheckForWordAndWrite(fin, fout, "liberty", "freedom", b);

        else

          fout.WriteByte(b);

    }

}

static void CheckForWordAndWrite(Stream si, Stream so,

    string word, string replace, byte first)

{

    int len = word.Length;

    long pos = si.Position;

    byte[] buf = new byte[len];

    buf[0] = first;

    si.Read(buf, 1, word.Length-1);

    string data = Encoding.ASCII.GetString(buf);

    if (String.Compare(word, data, true) == 0)

        so.Write(Encoding.ASCII.GetBytes(replace), 0, replace.Length);

    else

    {

        si.Position = pos;    // reset stream

        so.WriteByte(first);  // write orig byte

    }

}

Performance Monitoring

        Performance too is already present in .Net will find By

Tools >> Performance Tools

The Windows Performance Monitor does not solve performance problems of testing, gathering, analyzing, and modifying a system applies equally well to all aspects of performance, including memory utilization.

NHibernate for .NET

NHibernate is a port of Hibernate Core for Java to the .NET Framework. It handles persisting plain .NET objects to and from an underlying relational database. Given an XML description of your entities and relationships, NHibernate automatically generates SQL for loading and storing the objects. Optionally, you can describe your mapping metadata with attributes in your source code.

NHibernate supports transparent persistence, your object classes don't have to follow a restrictive programming model. Persistent classes do not need to implement any interface or inherit from a special base class. This makes it possible to design the business logic using plain .NET (CLR) objects and object-oriented idiom.

NHibernate key features:

Natural programming model - NHibernate supports natural OO idiom; inheritance, polymorphism, composition and the .NET collections framework

Native .NET - NHibernate API uses .NET conventions and idioms.

Support for fine-grained object models - a rich variety of mappings for collections and dependent objects

No build-time bytecode enhancement - there's no extra code generation or bytecode processing steps in your build procedure

The query options - NHibernate addresses both sides of the problem; not only how to get objects into the database, but also how to get them out again

Support for "conversations" - NHibernate supports long-lived persistence contexts, detach/reattach of objects, and takes care of optimistic locking automatically

Free/open source - NHibernate is licensed under the LGPL (Lesser GNU Public License)

Lets Start Coding :

Download NHibernate DLL and refer in Project

http://umn.dl.sourceforge.net/sourceforge/nhibernate/nhibernate-1.0.4.0.zip

NHibernate Create,Update,Delete Document [Simple example is explained]:
http://sdesmedt.wordpress.com/2006/06/13/nhibernate-part-2-crud-with-nhibernate/

NHibernate Create,Update,Delete SourceCode :

http://www.box.net/index.php?rm=box_v2_download_shared_file&file_id=f_13824980

[you will get a different Configurations for different database]
http://www.hibernate.org/361.html?cmd=prntdoc


Very Good Document :
http://www.hibernate.org/hib_docs/nhibernate/html_single/

Who is Online

Write Following Code Inside the global.asax



sub Application_OnStart
application("ServerStart") = now
end sub

function Decrypt(sText)
'Replace with your own decrypt ;)
Decrypt = sText
end function

function adZero(sText)
'Adds zeros to the beginning
if isNull(sText) then exit function
adZero = string(5 - len(sText), "0") & sText
end function

Sub Session_OnStart

'Session_OnStart is automatically called by the ASP Interpreter
'when a new session starts. (Probably a new visitor)

'TimeOut value determines the period in minutes when
'the session is assumed to be abandoned

Session.TimeOut = 10

'Get the current active users list
sActiveUsersList = application("ActiveUsersList")

'Get the new user name from the cookie
sNewUserName = Decrypt(request.cookies("devUserName"))

'If the visitor doesn't have a registered username, assign Guest(n) label
if sNewUserName = "" then sNewUserName = "Guest-" & AdZero(CInt(application("Visitors")))

'Initial action time
sLastActionTime = Time

'User info consists of user name, last action time, and the page viewed
sUserInfo = sNewUserName & "<|>" & sLastActionTime & "<|>" & sLastPageViewed

'Add this user to our collection with SessionID being the key
'(See the top of this file to see how dOnlineUsers object is initiated)
dOnlineUsers.Add Session.SessionID, sUserInfo

'Lock the application variable and update the contents
application.lock
'The number of active users
application("ActiveUsers") = application("ActiveUsers") + 1
'The number of visitors since last application reset
application("Visitors") = application("Visitors") + 1

'If the date is different than the previously stored date,
'it means we passed midnight, so reset our "today" counters
if application("TodaysDate") <> Date() then
application("PageViewsToday") = 0
application("VisitorsToday") = 0
end if
application("VisitorsToday") = application("VisitorsToday") + 1
'Store the date
application("TodaysDate") = Date()
'Unlock and go
application.unlock

End Sub

Sub Session_OnEnd
'Session_OnEnd is automatically called by the ASP Interpreter
'when the specified TimeOut period has passed after user's last action

on error resume next

'Remove this session from our collection
dOnlineUsers.Remove Session.SessionID

'And update the application variables
application.lock
application("ActiveUsers") = application("ActiveUsers") - 1
application.unlock

End Sub
</SCRIPT>

following code is used to display the list of the online users


on error resume next
response.write ""2"" cellspacing=""1"" border=""0"" width=""100%"">"
response.write""
aSessions = dOnlineUsers.Keys
for iUser = 0 to dOnlineUsers.Count - 1
sKey = aSessions(iUser)
sUserInfo = dOnlineUsers.Item(sKey)
aUserInfo =split(sUserInfo, "<|>")

sUserName = aUserInfo(0)
sLastActionTime = aUserInfo(1)
sLastPageViewed = aUserInfo(2)

if sUserInfo <>""then
iUsrCount = iUsrCount + 1
response.write""
response.write""
endif

next
response.write"

	Who	When	Where
"right"">" & iUsrCount & ".	" & sUserName & "	" & sLastActionTime & "	" & sLastPageViewed & "

"

Binding Datagrid values at client side to controls

< asp:DataGrid ID="dgdPackageDetails" Runat="server" width="100%" AllowSorting="False" GridLines="None" AutoGenerateColumns="False" >
< Columns >
< asp:TemplateColumn >
< HeaderTemplate >
Date
< /HeaderTemplate >
< ItemTemplate >
< a title="< %# DataBinder.Eval(Container.DataItem,"Date","{0:d}")% > " href="#" onclick="displayPackageDeatils()" >
< %# DataBinder.Eval(Container.DataItem,"Date","{0:d}")% >
< /a >
< /ItemTemplate >
< /asp:TemplateColumn >
< asp:TemplateColumn >
< HeaderTemplate >
Name
< /HeaderTemplate >
< ItemTemplate >
< a title="< %# DataBinder.Eval(Container.DataItem,"Name")% > " href="#" onclick="displayPackageDeatils()" >
< %# DataBinder.Eval(Container.DataItem,"Name")% >
< /a >
< /ItemTemplate >
< /asp:TemplateColumn >

< /Columns >
< /asp:DataGrid >

/***************************************************************************
Name : displayPackageDeatils
Parameters : None
Purpose : To display the contents of the grid in the respective controls
***************************************************************************/
function displayPackageDeatils()
{
var objForm=document.frmPackageInput; // frmPackageInput is the id of the form
objForm.txtDate.value=trimAll(event.srcElement.parentNode.parentNode.cells(0).innerText) //where txtDate is id of a textbox
objForm.txtPackageName.value= trimAll(event.srcElement.parentNode.parentNode.cells(1).innerText)//where txtPackageName is id of a textbox
}
/***************************************************************************
Name : trimAll
Parameters : The string that is to be trimmed
Purpose : To perform both left and right trim of a given string
***************************************************************************/
function trimAll(sString)
{
while (sString.substring(0,1) == ' ')
{
sString = sString.substring(1, sString.length);
}
while (sString.substring(sString.length-1, sString.length) == ' ')
{
sString = sString.substring(0,sString.length-1);
}
return sString;
}

Export Grid to Excel

Response.Clear();

Response.AddHeader("content-disposition", "attachment;filename=FileName.xls");

Response.Charset = "";

Response.Cache.SetCacheability(HttpCacheability.NoCache);

Response.ContentType = "application/vnd.xls";

System.IO.StringWriter stringWrite = new System.IO.StringWriter();

System.Web.UI.HtmlTextWriter htmlWrite = new HtmlTextWriter(stringWrite);

myDataGrid.RenderControl(htmlWrite);

Response.Write(stringWrite.ToString());

Response.End();

Applicatioon Settings

place this inside the configration setting present in the web.config file
----------------------------------------------






System.Data.Oledb.OleDbConnection con= new System.Data.Oledb.OleDbConnection();
con.ConnectionString=ConfigurationSettings.AppSettings["_dsn"];
con.Open();

.Net Snippets

Add Number to Date


App Settings


Export Grid to Excel

Add number to time

private void button1_Click(object sender, System.EventArgs e)
{
int dh1=System.DateTime.Now.Hour;
int dm1=System.DateTime.Now.Minute;
int ds1=System.DateTime.Now.Second;

int dh2=2;
int dm2=30;
int ds2=0;

int rh=dh1+dh2;
int rm=dm1+dm2;
int rs=ds1+ds2;

while(rh>=60)
{
rm=rm+1;
rs=rs-60;
}
while(rm>=60)
{
rh=rh+1;
rm=rm-60;
}
while(rh>12)
{
rh=rh-12;
}

textBox1.Text=rh.ToString()+":"+rm.ToString()+":"+rs.ToString();

}

private void button2_Click(object sender, System.EventArgs e)
{
System.Data.OleDb.OleDbConnection con = new System.Data.OleDb.OleDbConnection("provider=Microsoft.Jet.OleDb.4.0; data source=c:/i-scan/database.mdb");
con.Open();
string sql="select thour from ordinary where oid='1'";
System.Data.OleDb.OleDbCommand cmd = new System.Data.OleDb.OleDbCommand(sql,con);
System.Data.OleDb.OleDbDataReader rset = cmd.ExecuteReader();
if(rset.Read())
{

int dh1=System.DateTime.Now.Hour;
int dm1=System.DateTime.Now.Minute;
int ds1=System.DateTime.Now.Second;
MessageBox.Show(dh1.ToString());



string [ ] d2 = rset.GetValue(0).ToString().Split(':');

int rh=dh1+Convert.ToInt16(d2[0].ToString());
int rm=dm1+Convert.ToInt16(d2[1].ToString());
int rs=ds1+Convert.ToInt16(d2[2].ToString());

while(rh>=60)
{
rm=rm+1;
rs=rs-60;
}
while(rm>=60)
{
rh=rh+1;
rm=rm-60;
}
while(rh>12)
{
rh=rh-12;
}


string ltime;
if(dh1==0) dh1=12;

if (dh1>12 &&amp;amp; dh1<24)
{
dh1=dh1-12;
ltime=dh1.ToString()+":"+dm1.ToString()+":"+ds1.ToString()+":PM";
}
else
ltime=dh1.ToString()+":"+dm1.ToString()+":"+ds1.ToString()+":AM";

string lotime=rh.ToString()+":"+rm.ToString()+":"+rs.ToString();

con.Close();
con.Open();
MessageBox.Show(dh1.ToString());
sql="update ordinary set logintime='"+ ltime +"',logouttime='"+lotime +"' where oid='1'";
MessageBox.Show(sql);
System.Data.OleDb.OleDbCommand cmd1 = new System.Data.OleDb.OleDbCommand(sql,con);
cmd1.ExecuteNonQuery();
con.Close();
}
else
{
rset.Close();
con.Close();
}

}

Retail Solution By RFID

Radio Frequency Identification (RFID) is evolving as a major technology enabler for tracking goods and assets around the world—from the point of manufacturing through to the retail point-of-sale. Because of its potential benefits, many of the world’s largest retailers have mandated RFID tagging for pallets and cases shipped to their distribution centers by 2005. In the retail industry, this is likely to impact around 200,000 manufacturers and suppliers, and will fuel the global market for hardware and software to support RFID.

An RFID Overview

RFID systems consist of tags, readers and a range of applications to track, monitor, report, and manage items as they move between physical locations. These devices are supported by a sophisticated software architecture that enables the fast, or near time, collection and distribution of location-based information.

All information stored on RFID tags, such as product attributes, dimensions and prices, accompanies items as they travel through a supply chain or other business process. This information can then be scanned wirelessly by a reader at high speed and from a distance of several yards.

RFID Benefits

• Warehouse and distribution productivity
• Retail and Point-of-Sale productivity
• Out-of-Stock reduction
• Inventory management
• Shrinkage reduction
• Supply chain error reduction
• Capital Asset tracking and management
• Counterfeiting reduction
• Accounts Receivable efficiency
• Promotional execution improvement

RFID Costs and Considerations

• Tags
• Readers
• Software and integration
• Data Warehouse
• Business processes and systems integration
• Sufficient computing power
• Redundancy with existing barcode systems
• Problems with reading tags on liquid and metal items